Overcoming inappropriate apparent rotation of vehicle wheels in videos by image modification

ABSTRACT

Apparatus and method are described to overcome the perception in viewed video or other presentations containing moving vehicles that the apparent rotation rate of their wheels is frequently inappropriate for the vehicle speeds, which is caused by the stroboscopic interaction between the wheel spokes and the rate at which the images are presented. A currently preferred embodiment operates off-line under human control on a previously recorded video, identifying the wheels and replacing them with suitably altered images which are perceived to rotate at rates appropriate to the vehicle speeds; or to suppress inappropriate perception of rotation. The apparatus and method results in a new recording of the original video in which the wheels appear to rotate at the correct rate for all vehicle speeds. Alternate embodiments implement a similar but automated process operating in near-real time for operating in-line on streaming videos; and variations achieving the same ends.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of PPA 61/477,748 filed on Apr. 21, 2011 by the present inventor. It is a Continuation In Part of that PPA.

BACKGROUND AND PRIOR ART

The following is a tabulation of some prior art that presently appears at least partly relevant:

U.S. Patents U.S. Pat. No. Issue date Patentee 5,436,672 Jul. 25, 1995 Medioni et. al. 5,280,530 Jan. 18, 1994 Threw et. al. 5,808,695 Sep. 15, 1998 Rosser et. al. 5,892,554 Apr. 6, 1999 DiCicco et. al. 5,953,076 Sep. 14, 1999 Astle et. al. 6,993,184 Jan. 31, 2006 Matsugu 2,773,990 Dec. 11, 1956 Polhemus, et. al.

U.S. Pat. Application Publications Pat. No. Publication date Patentee U.S. 2008/0226253 Sep. 18, 2008 Steinberg et.al.

Search fields 348/571, 348/580, 348/583, 348/699

NON-PATENT LITERATURE DOCUMENTS

-   G. Medioni, G. Guy, H. Rom, A. François, Integrated Media Systems     Center, USC, Los Angeles, U.S.A., “Real time billboard substitution     in a video stream”

BACKGROUND OF THE INVENTION

This invention overcomes the frequent inappropriate apparent rotation rate of the wheels of moving vehicles that commonly appear in televised, video, filmed, computer, and other media images. This inappropriate rotation is caused by the stroboscopic effect, which arises because many wheels of moving vehicles, trucks, wagons and other vehicles, are composed of, or have on them, periodic or symmetrical structures such as spokes or wheel covers hubcaps with regularly placed areas, markings, or other designs, and are seen by the viewers only at the periodic and discrete intervals given by the rapid sequence, or the frame rate, of the video display medium. This stroboscopic effect, which is equivalent to the wheels or wheel covers being illuminated only at the rapid sequence of instants given by the frame rate, frequently makes the wheels appear to rotate at rates that are unrelated to the actual rate at which they should rotate due to the motion of the vehicle. Thus the wheels are frequently seen rotating forward, backward, or not at all, bearing little correlation with the actual vehicle speed. The same effect is also apparent when basically linear objects with regular features, such as ladders, are moved across the video field of view, the ladder appearing to have inappropriate linear motion, forward, backward, or appear motionless.

The origin of the stroboscopic effect is described referring to FIGS. 1 through 4, in which:

FIG. 1 illustrates a typical car, as example of a vehicle

FIG. 2 illustrates a typical wheel assembly on a typical car, as example of a vehicle

FIGS. 3A-3L show the inappropriate apparent rotation rate of a wheel at one vehicle speed

FIGS. 4A-4L show the inappropriate apparent rotation rate of a wheel at a different vehicle speed

These figures contain numerical elements as follows:

-   10 Vehicle (in FIG. 1) -   20 Wheel assembly (in FIGS. 1 and 2) -   21 Wheel assembly with distinctive imaginary mark on tire (in FIGS.     3A-3L and 4A-4L) -   22 Tire (in FIG. 2) -   23 Imaginary distinctive mark on tire (in FIGS. 3A-3L and 4A-4L) -   24 Rim of wheel (in FIG. 2) -   25 Stationary vertical reference line (in FIGS. 3A-3L and 4A-4L) -   26 Inner surface of wheel (in FIG. 2) -   28 Spoke structures of wheel (in FIG. 2) -   29 Hub or portion of vehicle holding the wheel (in FIG. 2)

This description begins with a discussion of a typical vehicle and the features of its wheels that give rise to the stroboscopic effect when video of the moving vehicle is presented to a viewer display, and which results in the inappropriate rotation perception problem. This is presented to facilitate understanding of the invention embodiments and their operation.

FIG. 1 shows a typical vehicle 10, including its spoked wheel assemblies 20, on the ground (which is not shown). FIG. 2 illustrates one wheel assembly, which generally comprises a rubber tire 22 mounted on a metal wheel. This metal wheel has a rim 24 and a generally cylindrical base 26 supported by spokes 28 or other structures, in this example shown as a six-sided star shape with truncated tips. This metal wheel connects the entire wheel assembly to the drive shaft or other mechanisms mounted on the vehicle suspension represented by housing 29 if the wheel assembly is driven by the vehicle's power train, or to surfaces not shown that are free to rotate if the wheel assembly is not driven by the power train of the vehicle. In some implementations the spokes of the wheel or some portions thereof are covered with a decorative plate or cap or hub cap. The particular design of the spokes or decorative cap or the like including the thickness, length, shape, color, and number of elements are unimportant except that there is usually an integer number of such spokes or design elements and they are generally regularly spaced around the wheel. It is this regular spacing of an integer number of elements which, together with their rotation rate and the images being presented to the viewer at another given rate, creates the stroboscopic effect.

FIGS. 3A through 3L illustrate the problem in a sequence of drawings of a rotating spoked wheel assembly as they would be seen if taken by a video camera and presented by the user displays at a fixed frame rate, in this example shown as 60 times per second, though the principle is the same at different frame rates. At each instant of time, in this example sequentially at every 1/60^(th) of a second, is shown the typical wheel assembly with six spokes but drawn in simplified form as a six-pointed star, as one of many shapes a wheel or hub cap or cover can take. Thus FIG. 3A is taken at the start of the sequence of video frames, FIG. 3B is taken 1/60^(th) of a second later, FIG. 3C is taken at 2/60^(th) of a second later, FIG. 3D is taken at 3/60^(th) second later, and so on with the last image, FIG. 3L, being shown taken at 11/60^(th) of a second later.

In each of these figures the tire of the wheel assembly has imprinted on it an imaginary mark shown as circle 23 so that we may discern and discuss the actual physical rotational position of the wheel assembly, which is now identified as 21 to differentiate it from the generic wheel assembly 20 which does not have such an imaginary mark or circle shown on its tire. It must be understood that this marking of the tire is purely for illustrative purposes, and that wheel assemblies or tires do not generally have such marks nor do they need any such markings in order for the stroboscopic effect to materialize. Also shown in these figures is a vertical reference line 25, also imaginary, so that the perceived rotation of the wheel assembly can be better visualized for the purposes of this discussion, since the actual spoked part of the wheel assembly in general has no identifying marks to distinguish one of its spokes or elements from another. It must also be understood that the wheel assembly is in contact with a road or other generally horizontal surface, which is not shown in any of the illustrations.

This wheel assembly 21 is rotating clockwise, as shown by the position of the painted reference circle 23 on the tire in successive images, the rotation being shown incrementally as about 50 degrees every 1/60^(th) of a second. The wheel assembly is generally rotating faster than the eye can track any mark on the tire. Thus the eye would not perceive the wheel assembly as rotating at all while it is actually rotating one full revolution in 7/60^(th) of a second, as shown in FIG. 3H. This is an equivalent rotational rate of 8.57 revolutions per second, which is 514 revolutions per minute. Given that a typical vehicle wheel assembly with tire has a diameter of 0.6 meters (2 feet) or so, this corresponds to an outer circumference of the tire of 3.14 times 0.6, which is 1.9 meters (6.28 feet), which means that the car and wheel assembly would move 1.9 meters (6.28 feet) linearly during the time a wheel assembly makes one revolution. Therefore a vehicle which rotates its wheel assembly at 8.57 revolutions per second travels 8.57 times 1.9 meters (6.28 feet) in each second, which is 16.3 meters per second (53.8 feet per second). This equates to 58.7 kilometers per hour (40 4 miles per hour), which is not an unusual speed for vehicles to be shown traveling in television, videos, or movies; and in any case is only shown as an example as the stroboscopic effect can and does occur at other speeds as well.

It is seen by examination of FIGS. 3A through 3L that while the actual rotation speed of the wheel assembly is 8.57 revolutions per second clockwise, the apparent rotation of the wheel assembly 21 is slowly counterclockwise, as seen by observing the triangular spokes, which are regularly placed around the wheel assembly, going counterclockwise past the static vertical reference line 25. In the example of FIGS. 3A through 3L the spoked wheel has apparently rotated 90 degrees in 11/60^(th) seconds, which equates to 1.36 revolutions per second counterclockwise. The eye is not capable of following the actual rapid clockwise rotation of the wheel assembly, and instead perceives that the wheel assembly rotates in a wrong direction by following the slow counterclockwise rotation. This effect is unchanged even if the reference mark were omitted, it being included in the illustration only for clarity.

Thus FIGS. 3A through 3L clearly illustrate the problem commonly experienced in video and film viewing, that often the perceived rotational speed of the wheel assembly bears little relationship to its actual rotational speed except for a few, fleeting instants of time should the vehicle be accelerating or decelerating through particular speeds when the stroboscopic effect produces the proper apparent rotation speed for the vehicle speed; and when it is stopped.

FIGS. 4A through 4L illustrate a different example in which the actual rotational speed of the wheel assembly 21 is still clockwise, but more rapid than that shown on FIGS. 3A though 3L. In this case the perceived rotation of the wheel assembly is zero as seen against the static reference vertical line because the wheel rotates an amount during 1/60^(th) of a second that brings the symmetrical wheel spoke pattern to a new rotational position which is such that it appears identical to the previous position even though rotated. This occurs because each spoke of the wheel is identical to all other spokes, and regularly located circumferentially around the wheel. This is a second example that the perceived rotational rates of the wheel assemblies on a vehicle often bear little relationship to their actual rotational speed, if seen in a video, movie, television, or other such display using rapid periodic presentation.

These inappropriate effects are disconcerting at best for the viewer, and deleterious to manufacturers and advertisers when displaying their vehicles and other vehicles for sale to consumers. While this application addresses both rotational and translational inappropriate apparent motion the application will principally discuss the rotational problem, it being understood that all aspects and content of this patent application and its embodiments apply to both rotational and translational effects. Remarkably this artifact has not been remedied, and inappropriate rotation images are unfortunately ubiquitous, despite the fact that automobile manufacturers alone spend hundreds of millions of dollars annually for television and video advertising, a substantial fraction of which feature moving vehicles yet all of which suffer from the inappropriate apparent rotation problem. Furthermore this problem is not new. Consider that film movie projection is well over 100 years old now and commercial television is over 60 years old, and both have suffered from the same stroboscopic motion effect all the time, from backwardly rotating wagon wheels in western movie films to ubiquitous televised images of vehicles with wheels whose rotation bears little or no relation to the actual speed of the vehicle, and frequently show stopped wheels, backward-rotating wheels, or forward rotating wheels at the wrong rate on a vehicle moving rapidly forward.

DISCUSSION OF PRIOR ART

A search was made of prior art, since the problem would seem to be susceptible to image manipulation via software to prevent or remove the effects of this stroboscopic effect, but no such image manipulation to this end was found in prior art, it apparently being accepted as an unfortunate but unavoidable fact of modern life. A large number of principally software-based approaches have been found in prior art that manipulate video images, many with real time effects. While most of these approaches aim to identify and replace portions of a live television image for displaying different signs and messages than present when the scene was imaged, principally for advertising purposes, it was initially thought that viable approaches to dealing with this problem surely exist in the prior art, yet no examples of such application were found. The salient prior art found is discussed below.

U.S. Pat. No. 5,280,530 to T. I. P. Threw et. al., Jan. 18, 1994 teaches a method of tracking a 3-dimensional object in 2-dimensional picture frames which relies on forming a template of the desired object, extracting a mask of the desired object, dividing the mask into sub-templates that are not associated with specific features of the object, and then searching subsequent frames to determine subsequent positions of the object. While Threw discusses image manipulation he does not teach any such manipulation of rotating images to alter their apparent rotation rates in video media as caused by the stroboscopic effect. Thus while Threw presents software tools that may be helpful they are far from sufficient and do not address the present problem.

U.S. Pat. No. 5,808,695 to R. J. Rosser, Sep. 15, 1998, teaches a method of tracking motion in a video image using a template to follow a set of landmarks to provide positional information which are then compared to a data table, and further requires a velocity prediction scheme for those landmarks. Rosser does not teach manipulation of rotating images to alter their apparent rotational rates as caused by the stroboscopic effect or any other effect, and does not discuss insertion of rotated images back into the scene at all. Thus while Rosser presents software tools that may be helpful they are far from sufficient and do not address the present problem.

U.S. Pat. No. 6,993,184 to M. Matsugu, Jan. 31, 2006, teaches a method of detecting the presence of and then extracting a specific object from an image at high speed, and altering the background of the image so that the same conditions surrounding the original object are duplicated, with intent to either identify the object image or to so alter a scene that the extracted object would not be missed. Matsugu does not mention or discuss changing the rotation of an object in the scene and reinserting the altered object back into the scene in near real time. Most fundamentally, Matsugu does not teach any manipulation of rotating images to alter their apparent rotation rates as caused by the stroboscopic effect. While Matsugu presents software tools that may be helpful they are far from sufficient and do not address the present problem.

U.S. Pat. No. 5,953,076 to B. Astle et. al., Sep. 14, 1999 teaches a system and method for replacing a given area containing a fixed sign in a video image with a different image by generation of a synthetic image of the area to be replaced, generating masks to occlude regions in the original image that are inappropriate for the image to be emplaced, and so constraining the replacement image that the substitution appears seamless. However, Astle does not discuss moving images, let alone rotating images, but discusses only static images such as signs in the background of a sports game. Most fundamentally, Astle does not anticipate manipulation of rotating images to alter their apparent rotation rates as caused by the stroboscopic effect. While Astle presents software tools that may be helpful they are far from sufficient and do not address the present problem.

U.S. Pat. No. 5,892,554 to DiCicco et. al., Apr. 6, 1999, teaches a method for placing and inserting an image into a live video scene which requires the establishment of landmarks in the scene and superimposing a coordinate system on it in order to track the movement of the image across the scene. Furthermore DiCicco requires the use of a separate video camera from the source camera to take auxiliary images of the scene in order to create a second scene, and identifying a second set of landmarks in the second scene for performing such insertion. DiCicco does not anticipate manipulation of rotating images to alter their apparent rotation rates as caused by the stroboscopic effect and in fact does not discuss substitution or insertion of rotated images back into the scene at all. While DiCicco presents software tools that may be helpful they are far from sufficient and do not address the present problem.

US Patent Application Publication US 2008/0226253 to Steinberg et. al., Sep. 18, 2008 teaches principally a method of recognizing a pattern in a video image for modification by resorting to a reference image and using a human recognizer, and then inserting a modified version back into the video. Steinberg does discuss rotation, but only as applied to rotating a video camera mounted on a tripod about a vertical axis, not to address rotating portions of images such as wheels; and he discusses applications aimed at imaging sporting events such as baseball games, which do not have rotating wheels in their images. Steinberg does not anticipate rectifying inappropriate apparent rotation of wheels and such caused by the stroboscopic effect. While Stenberg presents software tools that may be helpful they are far from sufficient and do not address the present problem.

U.S. Pat. No. 5,436,672 to Medioni et. al., Jul. 25, 1995 teaches a method for inserting a pre-determined fixed pattern into successive frames of a video image, following global motion. The intent of the patent is to substitute new text for the contents of a billboard visible in generally in sports events for the purpose of advertising. Medioni does not discuss rotation of image parts at all, and does not anticipate rectifying inappropriate apparent rotation of wheels and such caused by the stroboscopic effect. While Medioni presents software tools that may be helpful they are far from sufficient and do not address the present problem.

Professional papers and other media are represented by a paper by Medioni et. al: G. Medioni, G. Guy, H. Rom, A. François, Integrated Media Systems Center, USC, Los Angeles, U.S.A, “Real time billboard substitution in a video stream”. This paper treats the same topic as the subject for Medioni's patent. The intent of the teaching is for substituting new text for the contents of a billboard visible in generally in sports events for the purpose of advertising. It also does not anticipate or discuss changing the apparent inappropriate rotation rate of wheels and such in video streams. This paper is similar to many others in the field, and while Steinberg and others present software tools that may be helpful they are far from sufficient and do not address the present problem.

A search of advertising literature also failed to turn up any inkling of recognition of the problem, let alone any potential solution to counter the results of the stroboscopic effect. As a result of the search it is evident that neither has the problem of inappropriate wheel image rotation in moving displays been perceived, nor a solution conceived in prior art. The same conclusions are reached regarding inappropriate perception of translational motion in video displays.

SUMMARY

The embodiments of this invention each utilize apparatus and method to modify the images of rotating wheels on a vehicle as shown in video, television, movies, computer, or other media displays so that the perceived rotation of the wheels appears appropriate for any speed at which the vehicle is shown and seen traveling in the video, television, movies, computer, or other media displays. The embodiments identify the wheels of the vehicle that are visible at any particular time in the video, identify the areas or features of the wheels that are regularly placed and thus cause stroboscopic effects in conjunction with the frame rate of the video, determine the wheel rotation rates and shapes, define the actual rotation rate ranges that cause inappropriate but detectable apparent rotation rates, modify those wheel images so as to remove their features that cause the stroboscopic effect or otherwise modify them to prevent the effect from materializing in order to ensure that any apparent rotation rate that is perceived by the viewers is appropriate for the actual vehicle speed, and insert the modified wheel images into the appropriate images of the vehicle in the proper frames of the video data stream replacing the images of the original wheels or features. An alternative approach is to synthesize the desired images of the wheels, whether the same or different from the original wheel designs, and sequentially “morph” them and rotate them as appropriate and then insert the morphed or rotated frames into the video data stream.

The currently preferred and alternate embodiments all provide apparatus and method and differ principally in the degree and kind of human operator participation in and control of the computing machine process to attain the same desired results; in whether the process is off-line and intended for modification of a video and its recording to be used later, or whether the process operates in near-real time and while the video is streaming to the used displays; in the degree to which the actual or apparent rotation rates of the wheels must be determined in the process; and in the means for such determination. The currently preferred embodiment operates off-line on recorded video, is human-operated and machine executed, and its end product is another recorded video. The first alternate embodiment implements a similar process but operates in near-real time as an automated process, and is intended to operate in-line on videos being streamed to viewers. Second and third alternate embodiments implement some variations of the currently preferred and first alternate embodiments, but achieve the same ends. All embodiments result in modified video in which the wheels of the vehicles appear to rotate at the correct rates for all actual vehicle speeds when the video is viewed in the displays.

It is important to note that many of the process steps of each of the embodiments utilize computer software, or programs, with specific functionality to perform specific actions. It is understood that there exist and are in common use a number of software products, running on special or general purpose computation machinery that can perform some if not most, and perhaps all, of the necessary functions and operations called for by the various steps of the process to be described. However if existing software is insufficient, requires augmentation, or some new software would be desirable such software is relatively simple and could be readily written and coded by people normally skilled in those arts, since most of the process steps are relatively straightforward and uncomplicated compared to the extreme sophistication and complexity found in software today. In addition existing, modified, or new or different software may be needed to harmonize the operation of existing software to best interact with the operator, or for the operator to interact with the computer, and could also easily be programmed by persons normally skilled in the art.

All these existing, modified, or new software products, the choice of actual software programs, codes, or routines, and the algorithms used therein, individually or collectively, are generically referred to as software means hereinafter. The overall functions of the software means called for in this embodiment are described but not specified in detail because a number of different such means might function equally well. Neither are their source code or other coding details described as any programmer with reasonable skills in the art could readily generate them, or modify existing programs to the same end; and the very existence of many such examples is proof that that is the case. Thus these software products or means are not claimed as new. What is new and is claimed is their particular use to attain the highly desirable and useful end result, which is unforeseen in prior art.

It is understood in the above definition that software is a collection of instructions, algorithms, data, and programs that tell computing machine hardware elements what to do, how to do it, and when to do it whether the hardware elements are logic elements, switching elements, calculation elements, memory elements, computing elements, mathematical elements, geometrical elements, reading elements, recording elements, display driving elements, or other hardware. Thus all of the non-human functions described are machine hardware elements operating under the instructions and commands of software or humans or both, and are not mere written lists or abstract thoughts. All human functions, whether using observational, judgmental, manipulative, calculating, or decision skills, always result in action commands or actions that are executed by the computer (the machine). It is further intended and understood that the word “human” and the word “operator” and their plurals are interchangeable.

A large number of feature or pattern recognition, feature designation, motion tracking, feature manipulation, and object extraction and insertion software products exist and are in general use for manipulating images in many applications of still and video image generation and processing. These are commonly used to identify billboards or signs in scenes and to replace them with other billboards or signs featuring different text or content, generally for the purpose of advertising, but have many other uses for image modification in still and video media and other functions. An even greater number of software products exist are in common use for recognizing, extracting, or tracking images of faces or other objects in still or moving picture media, and are ubiquitous in photograph and video cameras, both for amateur and professional use. Such techniques can operate in real time or in delayed presentation, and some of these techniques were the subject of the patents cited as prior art.

In addition, a number of other software techniques also exist that break down an image into basic components and rearrange them at will to construct new images from them, or generate new images possessing some of the desired features of the original ones. Still others exist that are used to draw multiple images in electronic media that are incrementally and progressively changed so that when assembled into a medium allowing rapid sequential presentation such as a video or a movie they create moving and changing characters or scenes generically designated as animated films. A number of some software-based techniques also allow changing the aspect from which an object or scene is viewed or its perspective, and some also have the ability to manipulate such images creatively. This latter is commonly done in so-called “morphing” of images from one form to another in video, as well as in creating new subject matter for animated video presentation including cartoons and combinations of animated and real objects and beings.

The following is a tabulation of a number of examples of prior art or commercially available products which illustrate some specific techniques for implementing a number of software-based techniques that could be applied to some of the functions identified in this embodiment. These techniques include image recognition, tracking, and/or manipulation software, frame freezing or “grabbing” and selection software, image rotation and/or scaling software, measuring software, and others. While the specific examples listed below do not address the objectives and purpose or principal teachings of this application, they serve as easily accessed illustrations that the software art is well developed that can contribute to or perform specific functions as described in the process flow of the embodiments, whether using the specific techniques described or developing new techniques best suited to the identified tasks. These examples also serve as illustrations that very many persons normally skilled in the art of conceiving, designing, coding, verifying, and operating such software products exist. Thus all embodiments will refer to this example list whenever a software approach or program is discussed for implementing functions of the described processes.

This list includes,

For image recognition, feature tracking, and manipulation:

-   -   US2006/0115113A-1, Jun. 1, 2006; US2009/0141940A1, Jun. 4, 2009;         U.S. Pat. No. 3,264,993; US2001/0031067 A-1, Oct. 18, 2001; US         2011/0115945 A-1, May 19, 2011; US2006/0170769 A-1, Aug. 3,         2006; U.S. Pat. No. 7,983,448 B-1, Jul. 19, 2011; U.S. Pat. No.         7,454,037 B-2, Nov. 18, 2008; US 2005/0275723 A-1 Dec. 15, 2005;         U.S. Pat. No. 6,363,160 B-1, Mar. 26, 2002;         For frame freezing or “grabbing”, selection, image rotation,         and/or scaling, insertion:     -   U.S. Pat. No. 5,229,852 Jul. 20, 1993; U.S. Pat. No. 6,253,238         B1, Jun. 26, 2001; U.S. Pat. No. 5,808,695; U.S. Pat. No.         6,317,165 B1, Nov. 13, 2001; U.S. Pat. No. 5,109,278, Apr. 28,         1992; U.S. Pat. No. 5,568,600, Oct. 22, 1996; U.S. Pat. No.         7,113,194 B2, Sep. 26, 2006.         For measuring of angles and distances, and coordinate         transformation, image rotation, or image transformation:     -   Some elements similar to software commonly used in         computer-aided drawing programs such as Turbo CAD         (www.turbocad.com);     -   PAXIT (www.paxit.com/paxit/measurement.asp); or Boeckeler         (www.boeckeler.com/cms/index.cfm/path/17933/87457/);     -   or in public presentation and drawing programs such as Open         Office Impress or Open Office Draw (www.openoffice.org)         In addition many programs that would be applicable to a number         of aspects of the software tasks envisioned in these embodiments         are freely available from sources such as         http://sourceforge.net, http://opensourcewindows.org,         http://www.sagemath.org, http://directory.fsforg,         http://www.osalt.com, and many others.

Throughout all the embodiments the term “video” or “video media” or “video images” will be used, without limitation, to indicate moving image information-containing devices including those in electronic form such as solid state memory and those in physical form such as compact discs, video cassettes, and digital video discs. Throughout all the embodiments “displays” and “display devices” will be used to indicate moving image presentation and viewing devices including television, movies, screens, computers, cellphones, tablets, and other devices that facilitate or can display such videos. Throughout all the embodiments the term “wheel assembly” and its plural will generally be used whenever the discussion pertains to a vehicle wheel including its mounted tire, as well as their plural. The term “wheel” or its plural will generally be used when addressing strictly the metal wheel component or components of a wheel assembly or its plural, which may or may not also include metal or plastic covers or plates with decorative features, but do not include a tire.

Nonetheless, it must be recognized that the term “wheel” and its plural are commonly used in ordinary speech to refer interchangeably to the “wheel assembly” and its plural, and thus such references in this application may be inferred to be either, or the singular or the plural, as best indicated by the context of the text within which they are found. Furthermore, while the embodiments generally refer to a single wheel or wheel assembly when describing actions or functions, those same actions or functions could also apply to multiple wheels or wheel assemblies on a vehicle, without prejudice, unless specifically identified otherwise. In addition the spokes of typical wheels, which play a central role in the embodiments, are in some cases not structural but implemented in decorative wheel covers. The embodiments will henceforth refer only to “spokes” which should be inferred as applying to either structural spokes or distinct areas in wheels or in decorative wheel covers, interchangeably, so long as they are regularly spaced in angle centered at the wheel center, around the wheel periphery.

Throughout the discussion of the embodiments reference is made to the frame rate of a video. In these embodiments this refers to the time between image sequences in the video, and is used generically, without limitation to interlace scan systems, progressive scan systems, analog or digital systems, or other formats for capturing, recording, memorizing, or displaying videos. While a number of examples use 60 frames per second to illustrate the implementation of the embodiments, the embodiments operate as well and are as valid for other rates or other definitions of frame rate.

Furthermore, throughout the discussion whenever “stores” or “stores in memory” is described associated with a function it is understood that the memory is actually a part of the computer, whether centralized or distributed. In addition, whenever the discussion refers to a function performing some action, that action is understood to mean that the computer performs that action, rather than some specialized additional hardware, whether the computer physically consists of one unit or several separate units. Additionally when discussing the process flow the word “step” and “function” and their plurals are used interchangeably and their meanings are intended to be the same.

BRIEF DESCRIPTION OF ALL THE DRAWINGS

FIG. 1 illustrates a typical wheeled vehicle with its typical spoked wheels.

FIG. 2 illustrates a wheel assembly and parts thereof.

FIGS. 3A through 3L illustrate a typical sequence of images of a wheel assembly taken one video frame apart, showing its inappropriate apparently backward rotation rate.

FIGS. 4A through 4L illustrate a different sequence of video images of the same wheel assembly rotating at a different rate, showing its inappropriate apparently stopped rotation rate.

FIGS. 5A and 5B are simplified illustrations of the apparent rotation rate of an accelerating vehicle and the time intervals when the inappropriate apparent wheel rotation must be corrected.

FIG. 6A-6F illustrate a number of different spoke arrangements commonly used in vehicle wheels.

FIG. 7A-7B are more complete illustrations of the apparent rotation rate of an accelerating vehicle and the time intervals when the inappropriate apparent wheel rotation must be corrected

FIGS. 8A-8C are some strategies anticipated to be applied for apparent rotation rate modification.

FIG. 9 is the first of four process flow diagrams of the currently preferred embodiment.

FIG. 10 illustrates the appearance and disappearance of vehicles in a video as a function of time.

FIG. 11 is a plan view of the geometry of a typical vehicle and video camera.

FIG. 12 is a side view of a moving wheel assembly.

FIGS. 13A-13F illustrate the means used to determine vehicle average speed as a function of time.

FIG. 14 is the second of four process flow diagrams of the currently preferred embodiment.

FIG. 15 is the third of four process flow diagrams of the currently preferred embodiment.

FIG. 16 is the fourth of four process flow diagrams of the currently preferred embodiment.

FIGS. 17A through 17L illustrate a sequence of video images of the same wheel assembly as shown in FIGS. 3A through 3L, at a slow actual rotation rate, but now with correct apparent rotation rate as a result of operation of the currently preferred embodiment.

FIGS. 18A-18F illustrate the blurred appearance of different wheels, each with a different number of spokes as per FIG. 6A-6F, rotating at rates considerably higher than those of FIGS. 17A-17L, the rates all being too high to be detected by eye and thus resulting in blurred but non-rotating images.

FIGS. 19 and 21 describe the process flow of the first alternate embodiment.

FIGS. 20A-20D illustrate the timing of the functions of the first and third alternate embodiments.

FIGS. 22-24 describe the process flow of the second alternate embodiment.

FIGS. 25-26 describe the process flow of the third alternate embodiment.

DETAILED DESCRIPTION OF THE CURRENTLY PREFERRED EMBODIMENT

Figures in this embodiment: FIGS. 5A through 18F. Identifying numbers of lines, features, and process steps. Only process equipment and functional steps are shown in bold.

-   10 Vehicle -   20 Wheel assembly -   21 Wheel assembly including imaginary mark -   22 Tire -   23 Imaginary mark on wheel assembly -   24 Rim of wheel -   25 Imaginary vertical reference line -   26 Inner surface of rim of wheel -   28 Spokes of wheel -   29 Hub supporting wheel -   30 Relation between actual and apparent wheel rotation rate as a     vehicle accelerates from being stopped -   32 Relation between actual and apparent wheel rotation rates around     synchronism with the frame rate -   34 Portion of the slow actual rotation rate near zero that is     detectable by eye -   36 Portion of the high actual rotation rate around synchronism that     is detectable by eye -   38 Line specifying which wheel actual rotation rates need to be     modified -   40-46 Portions of the high actual rotation rates lines around the     sub synchronous points, which appear as lower and inappropriate     apparent rotation rates -   48-54 Portions of the high actual rotation rate lines around the sub     synchronous points, which appear as high and inappropriate apparent     rotation rates, or as non-rotating blurred images -   56 Portions of the high actual rotation rate line around one super     synchronous point, which appears as high and inappropriate apparent     rotation rates, or as non-rotating blurred images -   58 Portion of the high actual rotation rate that is detectable by     eye around the first super synchronous point -   62 Source of video -   64 Provision of vehicle and wheels data tables, graphs, data,     instructions, and assessment and decision criteria -   66 Provision of the needed software tools, programs, and manuals -   68 Provision of displays, controls, mice, keyboards, and plotting     aids for operator -   70 General Purpose or Special Purpose Computer with memory, logic,     and processors, and specific software programs -   72 First imposed relationship between the actual and apparent     rotation rates -   74 Second imposed relationship between the actual and apparent     rotation rates -   76 Third imposed relationship between the actual and apparent     rotation rates -   78 Fourth imposed relationship between the actual and apparent     rotation rates -   80 Fifth imposed relationship between the actual and apparent     rotation rates -   82 Sixth imposed relationship between the actual and apparent     rotation rate -   84 Seventh imposed relationship between the actual and apparent     rotation rates -   86 Truncated portion of 34 -   88 Extended portion of 34 -   90 Eighth imposed relationship between the actual and apparent     rotation rates -   92 Ninth imposed relationship between the actual and apparent     rotation rates -   94 Tenth imposed relationship between the actual and apparent     rotation rates -   96 First vehicle appearance in the video -   98 Second vehicle appearance in the video -   99 Likeness of vehicle -   100 Video camera -   102 First average wheel rotation rate -   104-113 Average wheel rotation rate points -   114 Piecewise linear vehicle speed curve as a function of time -   116 Smoothed vehicle average speed curve as a function of time -   118 Derived actual wheel actual Rotation rate as a function of time -   120 Identification, choice, and designation of a vehicle's     appearance in the video, and wheels -   124 Auto-tracking the vehicle, wheels, and supplying images -   126 Setting M frames, determining vehicle speed, direction, angles -   128 Provision of image rotation, coordinate transformation, and     measuring aids -   130 Selecting a wheel and determination of the wheel and wheel     assembly dimensions, angles -   132 Determination of the wheel average actual rotation rate for the     M frames, and storing -   133 Optional decision function regarding the accuracy of the     measurements -   134 Decision function on whether the wheel horizontal and vertical     dimensions are alike -   136 Expansion of the horizontal wheel dimension if required, to     match the vertical -   138 Counting the number of spokes or regular decorative features in     a wheel -   140 Command to display the choices of actual rotation rate/apparent     rotation rate graphs -   142 Retrieving and displaying the choices of actual rotation     rate/apparent rotation rate graphs -   144 Choosing the most appropriate actual rotation rate/apparent     rotation rate graph for the video's parameters -   145 Playing and displaying the video on command -   146 Identification of the actual rotation rate ranges that must be     modified -   150 Choosing an appropriate image substitution strategy -   152 Preparation of choices of modified wheel images for operator's     selection -   154 Selection of the most appropriate wheel image to substitute -   156 Decision function on whether new images are needed or to accept     current image -   158 Compression of the horizontal wheel dimension back to its     original value -   160 Command to 70 to step through the frames of the actual rotation     rate range and insert the images -   162 Insertion of the modified images into the video frames of the     selected wheel and range -   164 Command to collect the modified frames into a temporary video -   166 Preparation of a temporary video of the modified actual rotation     rate time range -   168 Observing the modified video -   170 Decision function on quality of the video -   172 Repetition of steps 150-170 one time -   174 Choosing a different actual rotation rate time range to modify -   176 Repetition of steps 150-178 for all frames in new time range,     and storing the data -   178 Choosing a different wheel of the same vehicle appearance -   180 Repetition of steps 132-178 for all frames in the time range,     and storing the data -   182 Choosing a different vehicle appearance -   184 Repetition of steps 120-182 for another vehicle -   185 Accumulation of all modified and unmodified frames into a new     video -   186 Displaying the new video for the operator's quality check -   187 Decision function on quality of the final video -   188 Making a recording of the new video and its distribution to     media

This embodiment envisions a process that relies substantially on the operator and his perception and skills, and is human controlled, however, every human function results in a command to the machine which then executes the command. The machine is comprised of a specific computer running specific software, algorithms, and instructions; and related input-output devices. The machine aids the operator by providing information and displays, making calculations and operations whether requested by the operator or self-determined, keeping track of the current step in the process with respect to all the steps of the process, and executing the operator's commands. This embodiment corrects rotation rate of the wheel images of vehicles in a video which are perceived in user displays as being inappropriate in direction of rotation or rotation rate for the actual vehicle speed, replacing them with images whose rotation either cannot be perceived by eye or retain some vestiges of rotational information, either of which are therefore appropriate for higher vehicle speeds, while leaving unchanged the images of wheels whose rotation rate is perceived by eye to be appropriate for the actual vehicle speed.

This embodiment is further oriented toward off-line functioning, such as processing a video recorded in electronic memory or in hardware form such as disc or tape, to correct inappropriate apparent wheel rotations in one or more vehicles appearing therein, and whose end product is another recorded video but with wheel rotations that appear appropriate for the vehicle speed throughout the video. This embodiment thus allows all steps and operations to be performed at leisure, and performed multiple times if necessary until the results are satisfactory, since the results are simply a corrected recorded video. This embodiment addresses serially the wheels of a vehicle and those of multiple vehicles in a video. The masculine gender will be used to refer to the human operator in all embodiments, recognizing that it is being done without any prejudice to or against female gender operators, but only for the sake of brevity.

The description and operation of this embodiment is shown in FIGS. 5A-B, 6A-6F, 7A-7B, 8A-8C, 9, 10-12, 13A-13F, 14-16, 17A-17L, and 18A-18F. References in the description will be made to functional blocks or steps or activities contained in the flow diagrams of the following figures, and such references will be shown in bold numerals without further identification. The process descriptions are in FIGS. 9, 12, 13, and 14, and clearly identify all the process steps envisioned, and those functions and operations which are envisioned as performed by humans and executed by the computer versus those envisioned to be performed purely by the computer in this embodiment.

This embodiment is designed to replace images of vehicle wheels in frames within a video, television, movie, computer, or other display which appear to rotate at speeds inappropriate to the vehicle speed. Since human viewers' eyes generally do not perceive motion if presented at repetition rates greater than about 10-15 times per second, blending or blurring all images above some such repetition rates, the modification of the video media to be recorded or modification of the video data stream need only occur at those times when the apparent rotation rate of the wheels is both inappropriate and perceivable as it is also below such a rotational rate.

In this embodiment the process of modifying the inappropriate wheel apparent rotation rate requires that the actual or actual rotation rates of the wheels of a moving vehicle be determined for a number of instants of time throughout the appearance of a vehicle in a video. Since the relationship between the actual and apparent rotational rates under stroboscopic viewing is well known and dependent principally on the number of spokes in the wheel and the video frame rate, wheel images with rotation rates appropriate for the vehicle speed at any time, or images blurred so as to appear to rotate at too high a rotation rate to be discerned, can be substituted for the inappropriate wheel images, resulting in a modified video with all wheel rotation rates appearing appropriate at all times.

This process is illustrated in FIG. 5A, which is a simplified plot to illustrate the principles of stroboscopic wheel rotation imaging. Referring to this figure the line 30 describes the wheel apparent rotation rate in revolutions per second for low actual wheel rotation rates when less than about 10 revolutions per second or so. The wheel is seen to be rotating at the same rate and in the same direction as the actual rotation rate, as measured by the value of the apparent rotation rate in revolutions per second, with a one-to-one relationship between the apparent rotation rate and the actual rotation rate. FIG. 5A also shows three lines of constant apparent rotation rate, labeled L1, L2, and L3. L2 is the line where the apparent rotation rate is zero—that is when the wheels are perceived to be standing still. L1 is the line of constant forward apparent rotation rate shown at a level of about 10 revolutions per second, in this example, below about which the wheel rotation is perceived as such and above about which the wheel is perceived as only a blur, without any rotation being detected. L3 is the line of constant negative, or backward wheel rotation, above which in a negative direction the rotation is blurred and not detectable by eye.

As an example, if a vehicle accelerates from standstill the wheel rotation proceeds along line 34 and is perceived as appropriate until it becomes a blur above a actual rotation rate of about 10 revolutions per second because the apparent rotation rate is above the blur threshold. As the vehicle continues accelerating, in this example, the apparent rotation rate continues to increase and not be detectable by eye until the actual rotation rate approaches about 50 revolutions per second, at which time the line 32 and 36 become dominant and the apparent rotation rate is seen to become negative, that is the wheel is perceived to be rotating slowly backward at about −10 revolutions per second, which occurs after the actual rotation rate exceeds about 50 revolutions per second. The lines 32 and 36 are straight lines with slope of one-to-one, as were 30 and 34, and is based on the frame rate of the video being assumed to be 60 fps, which results in the wheel being apparently stopped when the actual rotation rate becomes exactly equal to the frame rate, which is then exactly 60 revolutions per second. This is because at that actual rotation rate the images are presented at the same rate as the wheel is rotating, and so the image is always captured at exactly the same rotational position of the wheel regardless of its number of spokes or decorative features. Since most spokes or features in wheels are regularly spaced circumferentially around the wheel, the result will be that the spokes will exactly repeat and the wheel will all appear as though it is at standstill. That is the stroboscopic effect at synchronism.

As the vehicle actual rotation rate increases further past the synchronous 60 revolutions per second rate the apparent rotation rate increases in the proper rotational direction, but at improperly slow rotational rates as though it rotated below about 10 revolutions per second since the actual rotation rate is greatly above the rate for which rotation can be detected by eye. Once the actual rotation rate exceeds about 70 revolutions per second the rotation again becomes a blur and is undetectable by eye. This slowing, standing still, and reversal of apparent motion at inappropriate rotation rates and vehicle speeds is the heart of the problem which is being addressed by this embodiment. The stroboscopic effect thus produces inappropriate slowly backward-rotating apparent rotation rate, inappropriate standstill, and inappropriate slowly forward-rotating apparent rotation rates, all occurring at inappropriate actual rotation rates.

While the apparent rotation rate may be inappropriate at many values of actual rotation rate, it is only noticeable by eye during the actual rotation rate intervals described by lines 34 and 36. Thus it is the purpose of this embodiment to modify the apparent rotation rate during the times that the actual rotation rate of the vehicle wheels coincides with lines 34 and 36 so as to cause the apparent rotation rate to become perceived as appropriate for the respective vehicle speed at those times. This is indicated in FIG. 5B, in which the apparent rotation rates to be modified are defined with reference to lines 34 and 36. The lines designated as TRR-10 and TRR-11 define the lower and upper limits for which wheel image modification will be required, and are so indicated by line 38, since at all other actual rotation rates the apparent rotation rate is either appropriate or an appropriate blur. In this example, TRR-1 is defined by the upper blur limit of 10 revolutions per second, TRR-10 is defined by the lower blur limit of negative, or backward 10 revolutions per second, and TRR-11 by the upper blur limit of +10 revolutions per second, all of these being non-limiting examples.

The fact is that at actual rotation rates between TRR-1 and TRR-10 the wheel images have a positive and appropriate apparent rotation rate from line 30 but simultaneously also a negative and improper apparent rotation rate from line 32. Neither will be dominant and the perception of blur has no rotational aspect, so they do not represent a problem. However, the interval from TRR-0 to TRR-1 presents a more complex situation, because the apparent rotation rate is simultaneously perceived as proper in both magnitude and direction from line 34, and improper in both magnitude and direction from line 32. However, the image with apparent rotation rate between TRR-0 and TRR-1, which is slowest and most appropriate, will be the brightest, have the most discernible spoke images and will be seen as dominant, and the blurred apparent rotation rate will either be rejected by the eye or simply form a blurred haze over the proper apparent rotation rate in this overlap region. The transition between there two regions and effects may be more gradual than implied by FIG. 5A, whether the actual rotation rate is increasing or decreasing, but experience shows that at low actual rotation rates the eye will accept primarily the correct apparent rotation rate.

The picture of the situation described by FIGS. 5A and 5B is different for different frame rates that may be used by the video. For example a video frame rate of 30 frames per second will have the synchronous point at a actual rotation rate of 30 revolutions per second. Since the actual rotation rate at the synchronous point will always equal the frame rate, the wheel apparent rotation rate will be perceived as zero at that point. Therefore the horizontal axis of FIGS. 5A and 5B could be replaced by a differently calibrated axis to correspond to a different frame rate. The slope of the lines 30, 32, 34, and 36 will still be unity, and so the calibration marks on both the horizontal and vertical axes would have to be expanded by a factor of two to accommodate such a frame rate change, and the presentation. However, the levels L1 and L3 will still occur at about plus 10 and minus 10 revolutions per second, as the blur limit will be substantially unchanged with frame rate.

It is to be emphasized that the blur limit of the eye-brain of typical viewers generally varies between 5 and 15 revolutions per second, though it could be somewhat higher or lower. Thus the 10 revolutions per second values of L1 and L3 discussed and shown in the figures should be considered examples rather than specifications or hard limits. As a result FIGS. 5A and 5B could as well be drawn at a different scale where L1 and L3 are at a actual rotation rate of 5 revolutions per second, and the corresponding apparent rotation rate is also at 5 revolutions per second. Other values that 5 or 10 revolutions per second could be appropriate for L1 and L3, as the principles are the same regardless of the scale.

Unfortunately, the complete stroboscopic effect is more complex than that shown in FIGS. 5A and 5B due to the effect of different numbers of spokes or regular features of the wheels. FIGS. 6A-6F portray wheels with different number of regularly spaced spokes (or other regular features) of the wheels of a vehicle, varying from 3 to 8 spokes. Though the number of spokes in a vehicle wheel varies, they are usually regularly placed around the center or periphery of the wheel, which is essentially uniformly the practice in vehicle wheels, and which results in the full stroboscopic effect. All descriptions and comments in the remainder of this embodiment assume such regular placement of spokes or features. While the embodiment will work for less than regularly placed spokes or features, the result may be less-than perfect modification of the improper or inappropriate apparent rotation rate, however the effect itself will also be weaker due to the lack of regularity. A means of handling these less-than-perfectly regular spoke designs will be discussed presently.

Wheels with 3 spokes are illustrated in FIG. 6A and those with 4-8 spokes in FIGS. 6B-6F. Wheels in common use also contain a greater number of spokes, up to 20 or more, but are not shown as the principles are the same. Referring to FIG. 6A, which shows a three-spoke wheel 28 mounting a tire 22, with the three spokes regularly spaced around the wheel periphery. The spokes are identified as A, B, and C. Since there are three spokes the angular distance between any two adjacent ones, such as A and B, will be 360/3 degrees or 120 degrees. Assuming a 60 revolutions per second frame rate and a 60 revolutions per second actual rotation rate the wheel will make a complete revolution, 360 degrees, in one frame which is 1/60^(th) of a second. Thus the image of the wheel and its spokes will be identical to that shown in every frame, as the wheel position is identical at every frame and will appear stationary. However the wheel position will also be identical and stationary if the actual rotation rate is increased to twice the synchronous rate, or 120 revolutions per second, because exactly two revolutions of the wheel occur between video frames. This will also be actual for three times the actual rotation rate, etc., so that there will be super-synchronous actual rotation rate points at integer multiplications of the synchronous frame rate.

To complicate matters, in addition if the actual rotation rate is reduced below the synchronous actual rotation rate other sub-synchronous actual rotation rate points will also appear. Using FIG. 6A as an example, if the wheel is rotating clockwise and actual rotation rate is reduced so that the wheel makes exactly 120 degrees rotation less than before in one frame time, or 1/60^(th) of a second in the example, spoke A will appear in the position of spoke B but the position of the spokes will appear identical to the original position due to the regular spacing of the spokes, even though the actual spokes may not be the same ones since they are indistinguishable from each other. This leads to a sub-synchronous actual rotation rate at (360−120)/360 or ⅔ the synchronous actual rotation rate. By the same reasoning if the actual rotation rate is further reduced so that spoke A appears in spoke C's position in one frame time, or 240 degrees rotation less in 1/60^(th) of a second, the actual rotation rate is (360−240)/360 or ⅓ the synchronous actual rotation rate as the spokes again appear unmoved, and thus describe a second sub-synchronous actual rotation rate point. Thus for a three-spoked wheel there will be two sub-synchronous actual rotation rates: one at ⅔ and one at ⅓ of the synchronous actual rotation rate. This in addition to the synchronous actual rotation rate and an infinite number of super-synchronous actual rotation rates.

Putting practical numbers into the actual rotation rates, which for example portray a 0.67 meter diameter wheel assembly rotating at a actual rotation rate of 60 revolutions per second with a video frame rate of 60/second, imply that the vehicle speed must be 151.6 kilometers per hour, or 94.8 miles per hour. Thus for a frame rate of 60 images/second, the principal actual rotation rates of concern are at or below the synchronous rate of 60 revolutions per second, and the super synchronous points, though fully applicable, generally will be of less interest for surface vehicles which only rarely attain speeds of 303 kph (190 mph) or more in videos. However the sub synchronous points will all fall in vehicle speed areas of interest. This conclusion will hold for other numbers of spokes in the wheels, as the super-synchronous points are independent of the number of spokes in a wheel.

Using the four-spoke wheel design of FIG. 6B as a second example, if the actual rotation rate is reduced so that the wheel makes exactly 90 degrees rotation less in one frame time, or 1/60^(th) of a second in the example, spoke A will appear in the position of spoke B and the position of the spokes will appear identical to the original position due to the regular spacing of the spokes, even though the actual spokes may again not be the same ones, since they are indistinguishable from each other. This leads to a sub-synchronous actual rotation rate at (360−90)/360 or ¾ the synchronous actual rotation rate. By the same reasoning if the actual rotation rate is further reduced so that spoke A appears in the same position as spoke C in one frame time, or 180 degrees rotation in 1/60^(th) of a second, the actual rotation rate is (360−180)/360 or ½ the synchronous actual rotation rate as the spokes again appear unmoved and describe a second sub-synchronous actual rotation rate point. Similarly if the actual rotation rate is reduced even further so that spoke A shows up in the same position as spoke D in one frame time the actual rotation rate is (360−270)/360 or ¼ of the synchronous actual rotation rate. Thus for a four-spoked wheel there will be three sub-synchronous actual rotation rates: one at ¾ one at ½, and one at ¼ of the synchronous actual rotation rate. This is in addition to the synchronous actual rotation rate and an infinite number of super-synchronous actual rotation rates.

By deductive reasoning the different number of spokes in the wheels shown in FIGS. 6C through 6F, which portray wheels with 5 through 8 spokes respectively, will each result in sub-synchronous actual rotation rate points of (1−(m/n)), where m is an integer equal to or less that the number of spokes, and n is the number of spokes in the wheel. Thus for three spokes sub synchronous points appear at 0.67 and 0.34 of the synchronous actual rotation rate; for four spokes the points appear at 0.75, 0.5, and 0.25 of the synchronous actual rotation rate; for five spokes the points appear at 0.8, 0.6, 0.4, and 0.2 of the Synchronous actual rotation rate; for six spokes the points appear at 0.83, 0.67, 0.5, 0.33, and 0.17 of the synchronous actual rotation rate; for seven spokes the points appear at 0.86, 0.71, 0.57, 0.43, 0.29, and 0.14 of the synchronous actual rotation rate, and for eight spokes the points appear at 0.88, 0.75, 0.63, 0.50, 0.38, 0.25, and 0.13 of the synchronous actual rotation rate; and so on for wheels with more spokes. Thus, in general, the number of sub synchronous points will be one less than the number of spokes or regular features in the wheel.

The existence of any sub synchronous point at which the apparent rotation rate is zero also means that for smaller as well as larger actual rotation rates than the purely sub synchronous one the apparent rotation rate will be non-zero and vary with actual rotation rate in the same way that it does away from the synchronous point, that is the apparent rotation rate grows negatively below and positively above that point. Thus the picture becomes more complex than FIGS. 5A and 5B, and is illustrated in FIGS. 7A and 7B, which describe the apparent rotation rate as a function of the actual rotation rate for a five-spoked wheel and a frame rate of 60 per second, as an example of the more general case. FIGS. 7A and 7B contain all the information from FIGS. 5A and 5B, but with the addition of the lines describing the sub-synchronous mode behavior, and the partial inclusion of the first super-synchronous mode.

Thus FIG. 7A contains, in addition to lines relating the apparent rotation rate to the actual rotation rate for the principal low actual rotation rate mode represented by line 30 and the principal synchronous mode represented by line 32, line 40 representing the 0.20 mode with sub synchronous point at actual rotation rate=12 revolutions per second; line 50 representing the 0.40 mode with sub synchronous point at actual rotation rate=24 revolutions per second; line 52 representing the 0.60 mode with sub synchronous point at actual rotation rate=36; and line 54 representing the 0.8 mode with sub synchronous point at actual rotation rate=48 revolutions per second. As in FIGS. 5A and 5B, the portions of those additional lines which give rise to inappropriate apparent rotation rates are identified as 40, 42, 44, and 46 while the apparent rotation rate rates falling outside the extent of those lines will be blurred. The actual rotation rate values that represent the transition between being perceived as rotating and being blurred and not perceived as rotating are indicated as TRR-2 and TRR-3 for line 40, TRR-4 and TRR-5 for line 42, TRR-6 and TRR-7 for line 44, and TRR-8 and TRR-9 for line 46.

These actual rotation rate identifiers are related to the actual rotation rates ranges which will require modification and those that will not in FIG. 7B. The entire range between TRR-1 and TRR-11 is composed of apparent rotation rates that are both wrong in magnitude and some are also wrong in direction, and thus must be modified. The region between TRR-11 and TRR-12 consists entirely of apparent rotation rates represented by line 56 that are so large as to be blurred, and so will not require modification. The range between TRR-12 and TRR-13 (not shown) will require modification because the apparent rotation rates represented in part by line 58 are either too small, rotate in the wrong direction, or both.

The region between TRR-0 and TRR-1 is a special case. As discussed in the example above, though five improper or inappropriate apparent rotation rates will manifest themselves in addition to the proper magnitude and direction of apparent rotation rate of line 34, the latter will tend to have a greater image brightness and contrast. This is because the blurred image will of necessity be dimmer than the images of the spokes since the blurred image has the average of the brightness of the spokes and the usually dark spaces between them. Furthermore, the eye is vastly more sensitive to resolved moving images than to non-resolved moving images, even if of the same size and shape. Therefore this actual rotation rate region, in general, will not require modification though it is not labeled so in FIG. 7B, and any inappropriately rotating images in this region will only be fleeting and dimmer and may not be objectionable.

It must be pointed out that the transition from detectable to blurred in any of these lines, for example from 34 to 30 may be perceived gradually rather than abruptly, depending on many factors, and may influence the choice of an image modification strategy and implementation. In addition it is likely that the actual threshold levels L1 and L3 are themselves a function not only of the perception of the eye, but also of the actual number of spokes in the wheel being observed. It is therefore important to keep in mind that the values of the L1 and L3 levels are neither fixed nor absolute, but shown for example only as their values could be different from those shown and discussed in this or the other embodiments. To that must be added the caution expressed for FIGS. 5A and 5B regarding the limit of the eye-brain of typical viewers, which generally varies between 5 and 10 revolutions per second, though it could be somewhat higher or lower. Thus the 10 revolutions per second values of L1 and L3 discussed and shown in the figures should be considered examples rather than specifications or hard limits. As a result FIGS. 7A and 7B could also as well be drawn at a different scale where L1 and L3 are at an actual rotation rate of 5 revolutions per second, and the corresponding apparent rotation rate is also at 5 revolutions per second. Other values than 5 or 10 revolutions per second could be appropriate for L1 and L3, as the principles are the same regardless of the scale.

There are at least three types of time sequences in common use in typical videos showing a vehicle. In the first a vehicle could start from standstill and accelerate to some speed. In the second a vehicle could be shown initially at some speed, and then decelerate to a stop. In the third a vehicle could be shown at some road speed, which could be constant or varying, which it could do at different rates and times. The apparent rotation rate as a function of time is different in each of these types. In the first type, as the actual rotation rate increases, the apparent rotation rate of the wheels will be seen as correctly increasing until they become a blur as the actual rotation rate and apparent rotation rate pass through L1. As the actual rotation rate continues to increase the wheels will continue to be principally blurred, however as the actual rotation rate passes through the regions represented by lines 40, 42, 44, and 46 the strength of the stroboscopic effect could readily cause the apparent rotation rate to vary between the L1 and L3 levels, and the RR will be perceived as going from slowly backwards through a momentary stop, through slowly forwards, several times as the vehicle speed and thus the actual rotation rate increases.

When the actual rotation rate goes above 50 revolutions per second in this example, the stroboscopic effect will be the strongest since moving along line 36 as it will then be near synchronism, and the backward-to stop-to forward apparent rotation rate will be dominant. Further increases in actual rotation rate will produce principally a blurred apparent rotation rate as forward blur is mostly indistinguishable from backward blur, until the actual rotation rate nears the super synchronous point of line 58 when the backward-to-stop-to-forward rotation effect again dominates the blurred apparent rotation rate of line 56 above synchronism.

The second type of time sequence is the opposite from the first. As the vehicle slows from an initial high speed the actual rotation rate steadily decreases until it becomes zero. The sequence of apparent rotation rates will be the reverse of that in the first example, with the apparent rotation rate going from slowing forward-to-stopped-to-increasing backward apparent rotation rates several times. At the end the apparent rotation rate will be appropriate both in magnitude and direction as line 34 dominates and the wheels come to a stop.

The third type, in which the speed and therefore the actual rotation rate vary slowly between two or more values, will see a series of slowing-to-speeding up and speeding up-to-slowing apparent rotation rates, and apparent momentary stopping in between depending on the extent of the differences between the actual rotation rate values. Thus the apparent rotation rates could be similar to that of either the first or the second types and could contain one transition around a sub-synchronous point, a few, more than one, or none at all depending on the ranges of actual rotation rate values experienced. These apparent rotation rates are not only intuitively and mathematically correct, but not surprisingly also match those experienced during watching videos of moving vehicles having spoked wheels. This embodiment is designed to correct these inappropriate apparent rotation rates.

This embodiment discusses a number of strategies that could be utilized to eliminate the inappropriate and inappropriately perceived apparent rotation rate of the wheels of a moving vehicle in such videos. These strategies could be applicable to the other embodiments as well, and will be discussed further therein. Three such strategies are shown in FIGS. 8A, 8B, and 8C and are non-limiting examples, as other strategies could also exist and would be as applicable to the embodiment. All of these examples use a vehicle that starts at standstill and increases speed through at least the synchronous point, though the reverse or other speed/time profiles would be as applicable.

FIG. 8A describes a strategy in which the apparent rotation rate follows the actual rotation rate from zero to its intersection with L1, at TRR-1, at which time the wheel images are modified to be blurred. These blurred images are substituted for the original images in the video and used for all higher actual rotation rates at or above L1. They could be used at one level of blurring such as following line 76, or they could be imparted a degree of blurring such as generated by wheels that rotate faster as the actual rotation rate increases, in which case they would follow some line such as 74 or 72.

FIG. 8B describes a strategy which begins as does that in 8A until the apparent rotation rate crosses L1 at TRR-1, then substitutes uniformly blurred images until the actual rotation rate is above TRR-11 which results in inappropriate and detectable apparent rotation rate above the synchronous point, and then substitutes wheel apparent rotation rates which are blurred by various degrees of faster rotation with increases of actual rotation rate, in which they would follow some lines such as 80, 82, or 84.

FIG. 8C describes a third strategy in which adopts similar paths as did those in FIG. 8A but now identified as lines 90, 92, and 94. However they originate at an extension of 86 identified as 88 which extends beyond the intersection with L1 in order to start the blurring substitutions at a point well beyond TRR-1 in order to make sure that the apparent rotation rate is well clear of potential vestigial spoke rotation visibility and potential stroboscopic effects, however weak.

The choice of strategies could be made on the basis of the range of vehicle speeds, the likely number of spokes in the wheels, or other conditions likely to be encountered in the video, As an example, 8A might be most applicable in videos with high speed vehicles, 8B in videos with low speed vehicles, 8C in videos with a large number of spokes in the wheels, and so forth. Other strategies could include choosing 8A line 76 for embodiments such as the second alternate embodiment, in which the operator has vehicle speed estimates made only at a low actual rotation rate and only made by eye rather than by measurement, and so could not take advantage of a more complex strategy. The three strategies are not the only ones that are possible and probably effective, but they are a representative set, recognizing that others may be as effective, and so they should be considered as non-limiting examples of the principles. These strategies and choice criteria would be supplied in 64.

Referring to the process description beginning in FIG. 9 and continuing on FIGS. 14, 15, and 16, a source of video 62 is provided by the operator, containing the image sequences of the vehicle whose wheels may exhibit inappropriate apparent rotation rate. Data tables of vehicle types and their characteristics, common and/or usual wheel types and designs associated with specific vehicles, criteria for decision-making, a number of charts or their digital equivalents of apparent rotation rate as a function of actual rotation rate for several different wheel diameters and spoke designs for several different video frame rates are generated prior to beginning operations, data on criteria for replacing wheel images, and other data are provided by the operator in 64. The type and amount of such data will be understood to be discretionary as the process of this embodiment is enhanced by but can operate without some of these inputs if the operator can generate the data as needed or use his judgment to develop it.

Software products, programs, and tools that will be needed for the subsequent operations are provided by the operator in 66. While these will be discussed in more depth presently, one key software program will implement a system of message boxes that could be displayed on 68 for the computer to communicate a status message, and a count or identification of which step in the process the system is in. There could also be displayed an action box such as including a “yes”, “no”, “go back”, and “go forward”, or “go to step number _(——————)” menu for the operator to select which would command the machine 70 to execute those commands. In addition other commands and information displays could be accommodated. There could also be a process flow chart or set of charts to aid the operator in stepping through the process, which could also be displayed on 68 on command, whether in one shared display or on a separate display. These displays and action means or their equivalents are intended to be the principal command and executive interface between the human operator and the machine.

The inputs from functions 62, 64, and 66 are loaded into a computer 70, which can be a general-purpose or a special purpose computer, so long as it is capable of accepting, storing, recalling, and executing the particular and appropriate software programs and routines; operate in response to external or internal commands; and output its responses and results. This computer 70 is the heart of the machine-oriented activities, and it is functionally connected to and from the process steps in these flow diagrams, whether shown as connected in the figures or not. It is also intended that the human operator functions in these process flow diagrams are connected to the computer 70, and vice versa, whether shown as connected in the figures or not. One or more display units and any number of associated control keyboards, mice, overlays, or other input or output or plotting aids are provided in 68, and also connected to 70 with two-way communications. The video stream from the source 62 can be loaded into computer 70, or may optionally be accessed directly from 68 or from 70 and started or stopped at any point in time as desired in this flow, in response to commands from the operator or from the computer itself.

Once steps or functions 62-68 are completed the process begins with the operator, in function 120, viewing the video or some portion thereof, either directly from 62 or from 70, identifying the first vehicle appearance to be addressed and its wheels, and manually designating them in a command to 70. This appearance of the vehicle in the video is illustrated in FIG. 10, which shows an example of a video, its start as time A, the appearance of a vehicle noted as at time B and its disappearance noted as at time C and identified d as 96, a second vehicle appearance at time D and its disappearance as time E and identified as 98, and the ending of the video in time F. This description continues referring to the vehicle as though there were only one vehicle making an appearance in the video, but it will be understood that the process applies to all appearances by one vehicle and also to all appearances of any other vehicles, should there be more than one. Thus 96 could be the first appearance of vehicle #1 and 98 its second appearance, as an example.

Although the operator can choose any point in the video to start the process proper he could advantageously choose a point in time in which a vehicle makes its first appearance in the video, such as point A in FIG. 9. Alternatively he could choose a point in time at which he first perceives the wheels of the vehicle to have an inappropriate rotation rate for the vehicle apparent speed. The operator designates not just the presence of a vehicle but actually designates the vehicle itself, and one or both of its visible wheels via their coordinates, distance from the edges of the video frames, or specific characteristics of their image that would allow the computer to track them. This designation would be accompanied by a command from the operator to the computer to then initiate automatic tracking of the vehicle and its wheels in computer function 124, for which it could use a number of commonly available software products similar to that used in “face recognition” programs and “object tracking” programs, which are ubiquitous in consumer digital cameras, or more sophisticated professional recognition and tracking software also commonly available. Function 124 can also supply images, portions of images, frozen video frames, or other information as requested by 120 for display.

Next the operator, in function 126, determines the vehicle speed and direction of travel by measuring the motion of its image over a number “M” of video frames at or near the initially chosen times in the video. One of these measurement frames could coincide with the chosen initial vehicle identification time, but that may not be necessary so long as it is near that time due to the relatively slow-changing aspect of many vehicle images in common video streams. These frames are supplied on operator command either from the video source 62 or from the stored video by 70, with commonly available software which can “grab” or “freeze” the frames, separate them from the video stream, and present them as desired to 68. As a non-limiting example illustrated in FIG. 11, the vehicle 99 image positions in three frames are digitally introduced into a common larger image containing the images as well as a representation of the video camera used for recording 100, and their relative geometry.

It will be recognized that a minimum of two positions of the vehicle in time are required to obtain the average velocity between the points, however the greater the number and their spacing in time the greater the velocity accuracy, so three frames will be used by way of a non-limiting example. In this display the vehicle images could initially be sized small enough so that their position at the three instants of time represented by the three chosen video frames were mere points in order to facilitate accurate measurements, or some point on larger images of the vehicle could be used as a point reference to attain the same end.

In order to facilitate this display geometry and the desired measurements 70 can perform any requested necessary coordinate transformations, image rotations, and specified measurements using commonly available programs for such purposes in computer function 128 for display on 68 in a view normal to the horizontal plane or to the plane of the vehicle travel, or other suitable geometry. This display could be provided together with graduated measurement aids such as on-screen rulers, protractors or other angle measurement devices, or other on-screen measurement aids for the video image, as requested by the operator, to facilitate function the measurements. Alternatively the operator can supply physical overlays with inscribed rulers and protractors, or simply use those instruments themselves, to make these measurements directly on the screen of 68. If he so desires the operator can also print the images and use physical measurement aids to determine the desired measurements. The purpose of these measurements is determination of the vehicle range or distance from the video camera, its speed, and its direction of travel relative to the video camera line of sight, all of which will be used for determination of proper wheel rotation rates and other quantities necessary for the recognition and modification of inappropriate apparent wheel rotation.

The process of determination of these parameters can be based on the geometry of FIG. 11, which is an example of a plan view of the vehicle 99 and its motion. This figure lays out the vehicle position at three instants of time as seen by the video camera 100, with the distances, ranges, and all relevant angles identified. While these instants of time could beneficially be chosen during the time period when the operator perceives the vehicle wheels to be rotating at improper rates for the perceived vehicle speed, a different choice in number or instants of time could also be used and will be discussed presently.

It is noted that the only quantity that is assumed to be known a-priori in FIG. 11 are the approximate length of the vehicle and the precise video frame rate. Since the scale of FIG. 11 on the monitor screen is unknown, knowing the approximate length of vehicle 99 and the time elapsed between the time points t-1 and t-2, available from the frame rate, allows an approximate determination of the distance D₁ traveled by measuring how many vehicle image lengths, or fractions of lengths, fit between the time points. Since the make and type of vehicle to be experienced in the video is known a priory and made part of 64, including the vehicle length, the length of the vehicle will be known. In the case where it is not most vehicles of a recognizable type have approximately the same length, the accuracy of measurement will be limited by this approximate knowledge, but should suffice for the purposes of this embodiment. Alternate techniques to determine the distance traveled between t-1 and t-2 such as using a calibrated display or other techniques could be used and could be as, if not more, effective.

Knowing the time difference between the two frames at the known frame rate then allows the speed of vehicle 99 along the path D₁ to be determined from V₁=D₁/T₁, where V₁ is the vehicle velocity, D₁ is the distance, and T₁ is the time elapsed between time points t-1 and t-2 at the known video frame rate. The distance D₁ will be measured in terms of the vehicle lengths. Analogously the distance D₂ between time points t-2 and t-3 can be determined, and thus the vehicle speed V₂ between those time points. Only two time points are required to determine the vehicle velocity if it is constant and its path straight, and its direction relative to the video camera subtends an angle such as the angle “a” which is neither very small nor very large. However, such may not be the case and thus more than two time points, or a choice of different time points, or both, may be required for vehicle speed determination for various instants of time. This may be especially so if the angle “a” or one of the other angles in FIG. 11 is very small (such as less than a few tens of degrees) or very large (such as a few tens of degrees less than 180 degrees), or the time difference between t-1 and t-2, or both are particularly small.

The angles a, b, c, and d, e, and fin FIG. 10 can be measured directly on the display screen. Then, using the Law of Sines, the ranges R₁ and R₃ can be determined. If the angles c and f are small and difficult to measure accurately then the fact that the three angles of a triangle always equal 180 degrees enables determination of the ranges R₁, R₂, and R₃. While the range measurements may or may not prove useful they are discussed for completeness.

Measurement of the angle of vehicle travel with respect to the line-of-sight from the video camera will be a simple angle measurement on the display screen if the display is a plan view. Non-plan view presentations of the relative geometry are also possible and will also suffice but require the use of coordinate transformation or image rotation software, which is commonly available and presented by 128 to 126 on command. If no a-priory vehicle length data are available there are at least two options: The approximate vehicle dimensions can be obtained by scaling the dimensions of buildings, windows, doors, people, pets, trees, or other features of the scene whose dimensions are known or can be reasonably estimated. Alternatively should that prove difficult, such as might be the case if the vehicle is shown against an outdoors country scene with few or no landmarks of known size, the vehicle velocity could be estimated by its approximate size and the change in that size over a number of video frames, since the frame rate is known. It will be understood that the vehicle speed and direction determination can proceed over a larger or smaller number of video frames, or over video frames which are contiguous or widely separated in time. It will also be understood that there exist software tracking and designation programs that accomplish all the above autonomously, with but a few conditional inputs, and their use could also be invoked.

The choice of which frames to use for the estimation of the vehicle parameters of FIG. 11 could be done in a number of ways, one of which is illustrated in FIGS. 13A-13G. Referring to those figures the appearance of the vehicle, identified as a portion of 96, is shown in FIG. 13A with an illustrative set of video frames shown. It must be pointed out that there would usually be a much greater number of frames in a typical vehicle appearance, which could easily last tens of seconds or more, and thus could contain on the order of hundreds of video image frames. One strategy for frame selection for display in FIG. 11 is seen in FIG. 13B, in which the operator has selected F1, the first frame he sees as the first, F7 as the second, F13 as the third, and every 7^(th) frame thereafter during the entire video appearance. The choice of every 7^(th) frame is arbitrary and used only for illustration purposes. Not all the selected frames are so indicated in FIGS. 12A-13F, but if they were they would follow the same assignment strategy.

The vehicle speed determined using the geometry established by these frames in FIG. 11 would necessarily be an average over the time spanned, in this example over sets of 7 frames, or 7/60 of a second apart if the frame rate were 60/second. This is illustrated as points of average speed every 7^(th) frame in FIG. 13C, one point for every averaging set of 7 frames. The points are identified as 102, 104, 106, 108, 110, 111, 112, and 113. This series of points describes a vehicle speed profile taken at the average points. The points could be connected by straight lines to emphasize that no information was gathered in between the sampling frames, in which case the a piecewise continuous curve of speed as a function of time such as 114 would result. While this curve is continuous the information accuracy at any instant of time depends on its distance in time from the frames which defined it.

A more accurate speed curve could be readily generated by connecting the points of FIG. 13C with a curve, using linear or curve fitting approaches or software programs such as a least-squares or other curve fitting techniques. A typical result of such a curve-fitting process is shown by example that could be like 116, shown in FIG. 13E, which is a much better approximation to the actual vehicle speed. Computer function 129 is then commanded by the operator, through 70, to collect and store all the vehicle measurements he has determined for every set of frames used, and then to apply the averaging and curve-fitting process to determine the best fit curve of vehicle velocity as a function of time for the vehicle appearance and store it in memory. It must be pointed out again that the speed points, choices of frames, and other parameters and resulting in the curve of 13E are purely arbitrary, and shown only as an example.

Once the vehicle range, speed, and direction are known for the vehicle appearance in the video a similar process can be used to ascertain the angle of the wheels of the vehicle to the video camera line of sight, since knowledge of those angles is what is ultimately desired in order to devise the modifications to the wheel images, which is the objective of this process. If the vehicle is not turning at the time both the front and rear wheels of the vehicle could be assumed to be at essentially the same angles as the vehicle 99 in FIG. 11 with respect to the video camera. Since in general the vehicle will not be at exactly right angles to the video camera line-of-sight there will be some foreshortening of the circular wheel assembly images, which would make their horizontal dimension smaller than their vertical dimension. Should the vehicle not be turning and the ranges between video camera and vehicle be large this foreshortening will be substantially equal for the front and rear tires/wheels, and then the same modification of the video's wheel images can be applied to both front and rear wheels of the vehicle.

However, in the general case, the vehicle may well be turning at the time, or the interval of time chosen may be too great for such simplifying assumptions, or the range between vehicle and the video camera could be small, and thus the like dimensions of the front and rear wheel images would not be the same, and might have to be considered and treated separately in the process flow in the general case. Furthermore the time interval between the time points t-1, t-2, and t-3 could also be large, in which case the foreshortening of the wheel images could be substantially different for front and rear wheels whether the vehicle is turning or not. Thus, while not mandatory it is assumed that the front and rear wheels should in general be considered independently in the process flow. As a separate consideration the scale of the scene displayed on the monitor screen may have to be amplified to get the needed accuracy for the wheel assembly size and angle measurements, depending on the initial scale and the geometry.

The operator then determines, as the next step in the process in function 130, the actual angle of a visible wheel to the line-of-sight to the video camera for sets of frames such as used for the vehicle speed determination, though a different set could also be used if desired provided that a vehicle speed as a function of time curve such as 116 in FIG. 13E was determined. In addition, 130 also determines the wheel assemblies' horizontal (X) and vertical (Y) dimensions for each frame. This could be done using analogous measurement techniques to those that were used to determine the vehicle angles and distances, however a greater degree of image amplification may be required on the displays of 68 since the wheels are smaller objects than the vehicle. This display could be provided together with graduated measurement aids such as on-screen rulers, protractors or other measurement devices or on-screen measurement aids for the video image, as requested by the operator, to facilitate function 130. Alternatively the operator can supply physical overlays with inscribed rulers and protractors, or simply use those instruments themselves, to make these measurements directly on the screen of 68. If he so desires the operator can also print the images and use physical measurement aids to determine the desired measurements. The purpose of these measurements is determination of the horizontal (X) and vertical (Y) dimensions of the designated wheel or wheel assembly as seen on the designated vehicle.

It is necessary to determine both the horizontal and vertical dimensions of the wheel assembly or wheel, both to determine whether sufficient foreshortening exists that it will have to be removed, and to be able to accurately calculate the actual rotation rate of the wheel assembly. The graphical technique for determining these dimensions can be analogous to that used for vehicle measurement, but the image view should preferably be along the horizontal plane generating a side view of the vehicle and its wheels, so that the wheel images are basically vertical and thus their dimensions easily measured. Once obtained, these data on angles and dimensions are all commanded to be stored in 70.

The operator then commands 70 to proceed to 129 to determine the frame rate being used by the video, and to store it. This function could actually be accomplished at several other points in the process flow, and is shown as 129 for convenience. The operator then commands 70 to proceed to 132 to calculate the actual rotation rate of the wheel using its vertical dimension and the vehicle speed. This is easily done referring to FIG. 12, which shows a wheel assembly in the vertical plane. The actual rotation rate of the wheel assembly will be simply TRR=V/πD, where TRR is the actual rotation rate in revolutions per second, V is the vehicle velocity in meters per second, and D is the wheel assembly diameter in meters. Thus 132 calculates the actual rotation rate for every set of frames for which vehicle data was determined and stored, which could be discrete points or look like FIG. 13F, and stores the data.

The process then proceeds to 133 in FIG. 14 which is an optional function in which the operator assesses the accuracy of the data he has gathered, including vehicle speed, wheel angles, wheel dimensions, and wheel actual rotation rate for a selected number of frame sets at his discretion. This is so that if the quantities he has obtained do not meet criteria specified beforehand in 64, which were determined to assure that the image substitution is likely to be satisfactory, he can command 70 to send the process back to the beginning or to 126 and repeat the parameter determination using a different selection of frames, and/or their timing. As an example he could reduce the number of frames between the selected frames so that the average values determined were closer to the instantaneous values at all significant times. He could also alter the timing of the choice of frames so that the geometry is different and relevant parameters could be measured with greater accuracy than if the angles such as “a” were very small (or very large—close to 180 degrees). He could also use measures such as standard deviation and variance of the various measurements to determine accuracy. However, this step could be omitted once the system is checked out and the operator has gained sufficient experience to operate it satisfactorily.

The process then proceeds to decision function 134 in which the operator determines whether the (X) and (Y) dimensions of a wheel are substantially alike—that is whether the wheel is being imaged at about right angles to its axis. This is because if these measurements are very unlike each other there will be substantial foreshortening of the image and some operations to follow may be difficult to perform accurately. If that is the case the operator decides NO and commands 70 to proceed to 136 which expands the (X) measurement to match that of the (Y) measurement, making the image of the wheel substantially circular. If the decision was YES the operator commands 70 to proceeds directly to 138. An alternative to measuring the (X) and (Y) dimensions would be to measure the angle the wheel subtended to the video camera line-of-sight, which could be used in a like way to expand the (X) dimension by multiplying it by the cosecant of that angle. Another alternative would be to shrink the (Y) dimension to that of the (X) dimension.

The next step is to determine the number of spokes or other regular features in the now circular wheel images, and designate their number to 70, so that the proper choice of apparent vs. actual rotation rates graph can be made, and the proper wheel image modifications chosen and applied. In this embodiment this determination is done by the human operator in 138 by simply counting their number on a frozen frame which is presented on command to the operator's display by 70, and sending that number back to 70. Since the number of spokes or regular features in a wheel do not change during the vehicle's appearance, nor during any subsequent appearance of the same vehicle, this measurement need only be done once for each different vehicle, and the determination can be done at the most convenient time or set of frames for the operator.

However, the choice of frame for the determination of number of spokes must be chosen so that the apparent rotation rate is between the L1 and L3 limits in FIG. 7A, or he is likely to just see a blur and be unable to count the spokes. Therefore, if the wheel's spokes are not clearly discernible in the first vehicle set of frames viewed in the first vehicle appearance the operator must search the video until he can get a clear spoke count, at which time he can return to the start of the vehicle appearance. To that end the operator commands the computer to play the video at any rate or series of steps he desires from any starting point he desires until he has found clear spoke images, whether rotating or at standstill, counts the spokes, and commands 70 to store that number. The assumption is made that all the wheels of a particular vehicle have the same wheel pattern, as is the usual case, particularly in new vehicle advertisement videos. Should that not be the case the operator would repeat these process functions separately for the different patterned wheels. As discussed previously, for a number of reasons the process functions 136, 138, 140, and 144 may be performed separately for the steering and non-steering wheels, though they could be processed simultaneously if their (X) and (Y) measurements were nearly the same.

Alternatively this function 138 of determination of the number of spokes/pattern of the wheels could be accomplished in non-circular wheel assembly images without (X) or (Y) wheel assembly image dimension modification by the operator even if the wheel angles to the video camera are substantially different than 90 degrees, as long as the angle is considerably greater than zero or considerably less than 180 degrees, so that the foreshortening is sufficiently moderate so that a clear view of the spokes/pattern exists sufficient to permit their accurate visual counting. If that option were adopted functions 134 and 136, as well the eventual 158, could be skipped or omitted.

The process then proceeds to 140 in which the operator commands 70 to proceed to 142 to display on 68 the several actual rotation rate/apparent rotation rate graphs stored by 64, so that he can choose, in 144, the graph that most closely matches the current set of parameters including the frame rate and the number of spokes and designate that to 70. Alternatively the operator can simply request that the graph calculated for the proper frame rate and number of spokes be displayed for his approval or disapproval. In the former case the operator could choose the most appropriate from a number of graphs of wheel apparent rotation rate as a function of actual rotation rate, for different values of the number of spokes and different video frame rates, such as illustrated in FIGS. 7A and 7B, as each of these will be different. One option would be for sets of such graphs to be prepared ahead of time for many combinations of number of spokes, frame rate, and other parameters if appropriate, and supply them as part of 64, in which case the operator simply chooses the most appropriate one for his display. Another option would be for the computer to calculate and prepare a graph on command for the particular conditions he has determined, and present that on his display. Whichever option is selected the operator can readily see all the features that are illustrated in FIGS. 7A and 7B, but prepared and calibrated for the specific wheel dimensions, number of spokes, and video frame rate of the particular vehicle appearance in the particular video being processed.

The operator then commands 70 to proceed to 145 to step through the video and display it on 68, along with the actual rotation rate of the wheel correlated with the time of the frames being shown. That way the operator can watch the video at any rate he chooses, and in 146 mark those portions of the video in which the wheel actual rotation rate is in the “modify” region of the proper apparent/actual rotation rate graph chosen, corresponding to those regions of FIGS. 7A and 7B. This is in order to later enable the computer to make automatic image substitutions rapidly, once the operator has determined the times for those actions. The operator then commands 70 to store the time boundaries defining those regions in 70.

The process then proceeds to 150 in which the operator accesses the various image substitution strategies supplied by 70 from 64, such as illustrated in FIGS. 8A-8C, and chooses the most appropriate wheel image modification strategy. The operator would select from among the options using criteria pre-determined and supplied in 64, or he could use his judgment, and save the chosen strategy in 70. The simplest strategy would be to substitute every frame image of a wheel in those chosen that are in a “MODIFY” category of FIG. 7B with the same image of a wheel of the same type and size, but whose image is well blurred such as would be due to rapid rotation, corresponding to line 76 in FIG. 8A. Since the tire may have uneven coloration or texture, and could also have letters and numbers visible, the tire as well as the wheel, namely the entire wheel assembly, could be modified so as to be blurred. This strategy anticipates that as a wheel's actual rotation rate increases or decreases and crosses the L1 or L3 threshold the image would suddenly change from one where the spokes are detectable to one where no spokes are detectable and the wheel is a blur.

Alternative strategies could substitute modified images chosen or designed so that their apparent rotation rate is imposed to be that called for as a function of actual rotation rate commensurate with lines 34 and 72-74 in FIG. 8A, lines 34 and 80-84 in FIG. 8B, or lines 86 and 90-94 in FIG. 8C. Yet another alternative strategy would be to have a stored set of wheel or wheel assembly images for substitution, with different degrees of blur in each as though experienced at different actual rotation rates, each actual rotation rate being at a slightly different offset from a point at which a actual rotation rate/apparent rotation rate line such as 30 in FIG. 7A crosses the line of the level L1. The difference from the first strategy is that the eye may perceive subtle differences in different substituted images that appear to rotate at actual rotation rates too high to fully resolve the spoke patterns, but yet may be more natural looking as the vehicle speed and actual rotation rate vary at or near the boundaries of a “DO NOT MODIFY” rotation rate region such as at TRR-1 or actual rotation TRR-11 in FIG. 7B.

Whichever strategy is chosen the operator then commands 70 to proceed to 152 which prepares the pertinent choices for the wheel image modifications in accordance with whatever strategy decisions were made in 150, and stores them for presentation to the operator. The process for the actual preparation of the blurred wheel assembly images, or the blurred and rotated images, can vary so long as the product is, or are, wheel assembly images whose shape, size, and color match that of the original wheel assembly images except for the modified portion containing the spokes or design features which would no longer be discernible. One approach could be taking an actual image from a frame which is in a high “DO NOT MODIFY” actual rotation rate range of FIG. 7B. Another approach could be generating a graphic design of the wheel in rapid rotation, and imaging it. Yet another could be generating a series of such graphic designs, but with each at a different but rapid rotation. Yet a different option would be to take one of the wheel images and morph it so as to appear to be rapidly rotating. There are surely more options, though any one would probably accomplish essentially the same objective. There are software programs in common use that could readily accomplish any of them.

However the modification options are designed, 152 then sends a sample of each to operator function 154 in FIG. 15 so that he can make his choice from the visual or other choices offered as to which wheel image or images to use in the actual video frame modifications. He can assess these in conjunction with the selected apparent vs. actual rotation rate graph and strategy and proceed to 156 in which he decides whether the modified wheel images prepared by 154 and presented to him were sufficient or whether rework or new choices are desired. If his decision is NO he commands the computer to send the process back to 152 for a repetition loop. If he decides YES then he has decided that no new images are needed and has verified his choice made in 154, which he has already stored in memory in 70. If his decision in 134 was NO he then commands 70 to pass the process to function 158 which performs the reverse of function 136, and compresses the (X) dimension of the modified wheel image back to the (X) dimension of the original image's measurement in the video. If his decision in 134 was YES then no (X) expansion was done so no compression will be required, and he commands 70 to proceed to 160.

In process function 160 the operator commands 70 to proceed to 162 to sequentially step through all the frames of the selected time range one by one and insert the selected modified wheel images into all the frames in that time range of the original video, and stores them all.

The actual modification by 162 of each selected frame of the video with the new images can be accomplished in a number of ways. The modified image must be sized so as to match the dimensions of the original image and its color, texture, and degree of “blurriness” must be appropriate for the actual rotation rate being addressed. If it was designed as a new graphic image in rapid rotation its design of that pattern must be realistic and its depicted rotational speed must be appropriate for the desired effects. The modified wheel or wheel assembly image can then be simply “pasted” into the wheel or wheel assembly image after the original wheel image was “cut”, much as is done in computer presentation graphics programs in common use. Or the image can be appropriately “morphed” and integrated into the set of bits that represent the original wheel assembly image, thus replacing the bits representing the original wheel assembly. There are also other software techniques in common use that can accomplish the equivalent. The net result, however it is accomplished, will be a wheel assembly image for every modified frame in which the wheel assembly appears as the new, modified image rather than the original.

In performing these modifications 162 would step through the designated time range to be modified one video frame at a time, perform the modification, store the modified image frame, and proceed to the next frame, repeating the operation until done. When complete the process proceeds to 164 in which the operator issues a command 70 to proceed to 166 to collect all the modified frames into a temporary video and show it to him in 168 when done. The operator then observes the temporary video in 168, either step by step or straight through, so that he can make a quality assessment of the modification process. If the operator feels that the quality of the temporary video is lacking he decides NO in his decision function 170 and commands 70 to send the process to 172 to repeat steps 150-170 one time.

This repetition, which is a combination of machine functions and human functions executed by machine, will conclude with the process in function 170 again. Once the operator views the new temporary video in 168, if he now feels that the quality is acceptable he makes a YES decision in 170 and commands 70 to send the process to 174 in FIG. 16 and simultaneously sends a signal to 70 to accept the temporary video of the first chosen time range for the first vehicle appearance and to place it in storage. If he feels that the quality is still unacceptable he can command 70 to send the process to 172 again in the hope that it will be corrected, or he could elect to send the process back to an earlier step between 128 and 150 at his option. He can also command 70 to abort the entire process as something is clearly not working as intended, and start all over from the beginning by sending the process back to 120.

Once in function 174 the operator returns the process to 144, selects a different time range to be modified in the video of the vehicle during its appearance, should there be more than one, and commands 70 to proceed to 176 to cause a repetition of steps 150-174 for the new choice of time range, storing all appropriate data in 70. Once the process makes it successfully to 174 again he continues to choose time ranges in the vehicle appearance and commanding 70 to proceed to 176 to repeat its functioning for every time range until there are no “modify” time ranges left. He then proceeds to function 178 in which he commands 70 to send the process back again, but this time to 130 to choose a different wheel assembly of the same vehicle should there be reason to do so, and commands 70 to proceed to 180 to cause a repetition of steps 132-178 for the new choice of wheel, including possible repeat loops and operator decisions as before, and data storage in 70. He then commands 70 to proceed to 182 in which he commands 70 to return the process to 120 and commands 70 to display the original video again in 124, from which he selects a different appearance of the same vehicle, if any, for processing or a new appearance of a different vehicle, if any. He then commands 70 to send the process to 184 which coordinates stepping through of the entire process for the new vehicle appearance, from 120 through 182.

This process is repeated until all the time ranges requiring modification for all the wheels in all the appearances of all the vehicles have been modified, and their quality accepted, at which time he commands 70 to proceed to 185 to assemble all the modified and all the unmodified video frames of all the time ranges addressed, for all wheels addressed, and for all the vehicle appearances, into a properly sequenced new continuous video.

The operator then proceeds to a final quality check in 186 in which he is shown the complete final video stream from 185, on his command. He then makes a decision in 187 as to its overall quality. If acceptable, using criteria from 64 or his own judgment, he decides YES which commands 70 to proceed to 188 to make one or more recordings of the modified video in one or more media types and formats, which are then ready to be made available for distribution to viewers instead of the original video 62. The recording(s) can be made as software, if such is the medium to be distributed, or if hard media by any machine capable of that task, whether it includes 70 or is simply turned on by 70. If the operator deems the final quality unacceptable he can command 70 to send the process back to 120 or another step to repeat the process, or simply abort. Whereas the default decision of 170 should be YES so that the process can be given more chances to perform as intended, the default decision in 187 should be NO so that a poor quality product will not make it all the way to a final output without a conscious decision by the operator to do so.

The result of the modification of the wheel images of this process will be wheels perceived to be correctly rotating for every time range when the vehicle speed is low, as shown in FIGS. 17A-17L, in which a wheel is seen as rotating appropriately for its actual rotation rate. These figures show precisely same wheel assembly 21 with the same imaginary reference circle 23 on the tire, rotating at the same rate, and shown at the same precise intervals of time as were shown in FIGS. 3A through 3L. In this case, however, as seen against the same stationary imaginary vertical reference marks 25, each wheel image at each instant of successive time at the same frame rate has the original image replaced by a new image whose rotational position is appropriate for the actual wheel rotational speed. Therefore the sequence of images in FIGS. 17A through 17L now has the wheel assembly shown to be always in the proper orientation that it would have were it to rotate in the video at the same rotational speed and direction as the actual physical real wheel assembly. The same result will be obtained by image modification of the conditions of FIGS. 4A through 4L even though the actual wheel rotation rate is different than that of FIGS. 3A through 3L.

When the actual rotation rate is not low, that is when it is above TRR-1 in FIG. 7A-7B, the wheel images are always deliberately blurred so as to prevent the spokes or other patterns to be perceived as rotating at all, which is appropriate at speeds above the blur limits of L1 or L3 in FIGS. 7A and 7B. This effect is illustrated in FIGS. 18A-18F, in which the likeness of rapidly rotating; blurred images of the wheels are replaced with concentric dashed lines which are circumferentially similar if not identical, and have no spoke detail discernible. While this would clearly eliminate all inappropriate apparent rotation rate perception the option exists for including some differences in the modified images of FIGS. 18A-18F in their degree of blurriness, or in some vestigial pattern that is dimly or subliminally perceived by the viewers in the interest of more realism as vehicle speed changes.

Thus the inappropriate apparent rotation of the wheels will have been remedied by image substitution where appropriate for all wheel actual rotation rates and therefore for all vehicle speeds, and the undesirable effect of the stroboscopic effect will have been completely eliminated; the image modifications and substitutions are undetectable by the viewer; and the process has produced an entirely new and unexpected result: the absence of apparent inappropriate rotation rate of wheels that do not correspond to actual vehicle speed as seen in video displays of moving vehicles. It will be understood that a case in which the actual rotation rate of the wheel assembly is in-between those exemplified in FIGS. 3A through 3L and FIGS. 4A through 4L, or rotating at greater or slower speeds, or containing wheel images with fewer or greater number of spokes or patterns would yield intermediary results all of which would be properly corrected for their respective conditions.

Since the process is off-line and thus not real-time the iterations can be repeated until the operator is satisfied that the proper quality product has resulted from the flow process of this embodiment, and that the inappropriate apparent rotations of the wheels in the video have been adequately modified and now appear correct. While the steps of the process in this embodiment will accomplish its objectives, some steps may not be necessary and might be omitted, while some may be combined into fewer steps. An example is the omitting of steps 134, 136, and 158 which made the wheel image circular for easier measurements and operations, and then made it non-circular again to match the original wheel shape. An option exists to perform all operations on the non-circular wheel images, which will be elliptical, since the are software programs that can perform the counting of the spokes and wheel sizing operations on elliptical images rather than on circular ones. While this would save some complexity and time there are limits to the ellipticity that can be accommodated in such operations due to the inherent noise and inaccuracy involved in measurements on small quantities which could decrease the accuracy of the image substitution process, and thus such techniques could be used providing the angle of viewing of the wheel with respect to its axis is not close to zero or 180 degrees.

Also some steps identified as machine steps might be more easily performed by the operator, or vice versa, there being a number of alternates to this embodiment using fewer or greater number of steps, or exchanging human and machine steps, which would be similarly effective while still maintaining a similar degree of control of the human operator over the process and its results. Thus it is understood that such possible variations are recognized and intended to be fully within the scope of this embodiment. It will also be understood that when human decisions or actions are identified in the process flow they are either commands to the computer to perform specific functions, observations that result directly in such actions without commanding the computer, or responses to requests by the computer for actions or decisions, all of which are human functions executed through and by the computer, and also intended to be fully within the scope of this embodiment.

The new images can also be made different from the original if desired, such as featuring a different wheel structure, pattern, shape or color, if desired. In fact the embodiment also would permit specific rotational or stroboscopic effects to be imposed on the final video stream to deliberately create some wheel apparent rotation speeds, whether constant or variable, or lack of them to replace the proper or otherwise apparent rotation speeds; or create new patterns that could change in time and space and could cause images or patterns to move within the wheel radially or circumferentially or both, should specific effects be desired. In fact it is quite possible to generate specific images of people or products which could be animated and appear instead of the spoked wheel images, and be used for advertising, artistic, or other purposes.

This process can operate regardless of the speed of the vehicle or the particular design of the wheels, wheel covers, hub caps, or other structure and thus, unless specifically so desired, can effectively create apparent final wheel rotation rates that are appropriate whether the vehicle is accelerating, maintaining a given speed, or slowing down. The wheels would also be seen to stop rotating when the vehicle speed is reduced to zero.

Detailed Description of the First Alternate Embodiment

This embodiment is an apparatus and method for a fully automated process implemented on a specific computing machine which is intended to operate in near-real time and in-line with streaming video services, with optional and then limited human control of the machine. This embodiment corrects rotation rate of the images of wheels of vehicles in a video which are perceived in user displays as being inappropriate in direction of rotation or rotation rate for the actual vehicle speed, replacing them with images whose rotation in viewer displays cannot be perceived by eye and therefore are appropriate for higher vehicle speeds, while leaving unchanged the images of wheels whose rotation rate is perceived by eye to be appropriate for the actual vehicle speed. This embodiment addresses the wheels of a vehicle and multiple vehicles in a video serially.

Figures in this embodiment: FIGS. 5A-18F (previous embodiment) and 19, 20A-20D, and 21.

-   190 Detecting a vehicle and its wheels, autotrack -   192 Manually identifying a vehicle or start process -   194 Selecting timing and strategy, set M frames and target frame -   196 Providing image rotation and measurement resources -   198 Determining vehicle speed, direction, angles, ranges -   200 Determining video frame rate and count number of spokes of one     wheel -   202 Determining the vertical and horizontal dimensions of a wheel -   204 Making a decision as to whether the wheel dimensions are about     equal -   206 Expanding the horizontal dimension until it equals the vertical     dimension -   208 Determining the wheel actual rotation rate over a set of N     frames -   210 Choosing an actual/apparent rotation rate graph and determining     if the frames are in a “modify” region -   218 Optional human review -   220 Decision function as to whether the frames require modification -   222 Choosing an appropriate image substitution strategy -   226 Selecting a modified wheel image from the available choices -   228 Decision function as to whether 204 was YES -   230 Compressing the horizontal dimension back to its original length -   232 Adjusting the size of the wheel modification image -   234 Insertion of the modified wheel image into the target frame -   236 Optional visual quality check -   238 Decision as to quality -   240 Decision as to timing and quality -   242 Repetition of functional steps for another wheel on the same     vehicle -   244 Repetition of functional steps for another vehicle appearance -   246 Optional accumulation and assembly of frames into a new video -   248 Optional operator visual quality checks -   250 Making the final quality decision -   252 Provision of fast video switch -   254 Distribution of the video stream to users

The process begins and proceeds throughout this embodiment under the control of 70, which implements all the functional steps discussed and shown in the process flow diagrams. Te process starts with computer function 190 in FIG. 19 with the computer software accessing the start of the video, which is illustrated in FIG. 10 as point A. 190 then autonomously identifies, designates, and begins tracking the vehicle's first appearance in the video, such as shown in the illustration of FIG. 10 as the point B of the vehicle's appearance, 96. This function would use object recognition software similar to the ubiquitous “face recognition and tracking” software in common use in cameras or “shape or feature identification” software in common use in billboard or advertising message recognition and substitution programs. Any of these or similar programs would operate on the vehicle shape, features, or its wheels. It is anticipated that the first appearance of a vehicle in the video stream would be used to start the process, though any appearance of any vehicle could also be used, keeping in mind that in this embodiment the inappropriate apparent rotation rate of the wheels can only be modified forward in time from the time of vehicle identification and track.

An option is provided in function 192 for the operator to identify and designate the vehicle to 70 while viewing the video 62 on 68 as the first step in the process, and is provided as insurance against the process having unexpected difficulties at its outset. Alternatively the operator can simply initiate the machine process by a “start” command to 190. The operator could also initiate the process by an even simpler switch actuation which would power or alert 70 to the fact that its functions are now desired, so as to prevent it from operating all the time on a video feed and rather only when and if so desired. Nonetheless, as it is intended that the process be automated in normal operations so as to minimize processing time delays, it is intended that 70 and the process starting with 190 to be active all the time. This could be implemented by 192 once the automated vehicle recognition function works reliably enough and has a low inappropriate start rate, as would be determined by analysis, experience, or both.

Once the process is started 190 begins observing the video image stream to detect and identify the first appearance by a vehicle in the video stream, following the instructions, data, and criteria of 64, and designates the vehicle by a unique identifier code. The function then similarly detects, identifies, and designates its wheel, or both wheels should they both be visible, and begins tracking the vehicle and each wheel separately. Function 190 also continually observes the video stream to detect if the vehicle makes more than one appearance, when it disappears, or when a second vehicle makes its appearance and disappearance, shown as point D and curve 98 in FIG. 10. Function 190 stores each vehicle and the wheel's identification in memory. Since the two visible wheels of the vehicle may have a different shape when the vehicle is turning, and since a second vehicle may make an appearance simultaneously with the first vehicle, it may be necessary to track and modify up to four different wheels in essentially the same near-real time period.

The process then proceeds to 194 which selects one of the vehicle's wheels and sets the timing strategy for the embodiment. This strategy has options, illustrated in FIGS. 20A-20D, keyed to the frames which appear at the video frame rate as a function of time, shown in FIG. 20A, and beginning with T₀. FIG. 20B illustrates the time that various functions in this embodiment require for their operation, beginning with the time required to designate the vehicle, labeled as DV; followed by the time required to designate the wheels, labeled as DW; followed by the time required to determine the vehicle speed, labeled as M; followed by the time required to determine the wheel assembly dimensions, labeled DD; followed by the time required to determine the number of spokes in the wheels, labeled as N; and lastly the time lag for the process to step through all the functional steps in the process, labeled as S.

The length of these functions as shown are purely as examples, and could require many more frames or fewer frames in actuality. If these steps are executed serially they will appear as shown in FIG. 20B and the total delay indicated as T₁, which would be the first frame that could be modified by the process stepping though its steps in this embodiment. However, the designation of the wheels and the measurement of the vehicle speed could start in parallel as they are not dependent on each other, and the measurement of the number of spokes could start independently also. The counting of the number of spokes can also occur in parallel with the vehicle speed measurement, but the wheel dimension determination must follow the vehicle speed determination, and the system total processing time must necessarily be additive to all the other functional times, so that the total time could be reduced to T₂ in FIG. 20C by paralleling some functional steps. In addition, once the process has stepped though and modified the first frame image, the process for doing so for the following frame images can be considerably shorter as long as it is within the same vehicle appearance. This is because only the vehicle speed and the wheel dimensions must be measured, and with the unavoidable system time delay the total delay could be shortened to T₃ in FIG. 20D.

The functioning of 194 requires that the total system time delay be measured and stored, which would be done by the equipment manufacturer, and possibly calibrated in the installation by the operator during the set-up process. As a result of the foregoing 194 selects all the process steps to occur serially for the next video frame if the vehicle appearance is new, or instructs that steps 196, 200, 210, 222, 226 and all of 190 except its scanning for the presence of the vehicle be skipped if the vehicle appearance is continued from the previous frame.

The process then proceeds to 198 to determine the vehicle speed, direction, and angles which will be needed in order to determine the wheel actual rotation rate, which in turn is necessary for the image substitution process. The process of determination of these parameters can be based on a geometry similar to that of FIGS. 11 and 13A-E and using frame designations and implemented by a process similar to that previously described for the currently preferred embodiment in, except that all steps are accomplished by the computer rather than by the operator through the computer, and aided by 196 which provides image rotation, coordinate transformation, and measuring software resources to accomplish its task. While the precise length of the vehicle will generally not be known in advance, this being a real time embodiment intended to be on-line in a video feed all the time, the computer uses estimates of vehicle lengths supplied in 64, but it may be possible to determine a more precise measure of the length by comparison of the vehicle, seen from the side or above by coordinate transformations or rotations, with elements in the scene background whose size in known, such as flagpoles, buildings, people, or other fixed or moving objects. While the accuracy of determination of the vehicle velocity will be limited by the approximate length knowledge, it should suffice for the purposes of this embodiment.

Once the vehicle speed and direction are known for the vehicle appearance in the video the process proceeds to function 200 which determines the video frame rate and counts the number of spokes in the wheel. The computer obtains the video frame rate by simply counting the time elapsed between video frames, which could take as little as two frames, and stores it. Determination of the number of spokes requires a finite period of time. To obtain the number of spokes the computer accesses the digital data representing a wheel assembly and wheel image, and scans the digital data in a pattern equidistant from the center of rotation of the wheel, or of the centroid of the wheel image, whichever is easiest to implement, and counting the number of times that the magnitude of those data exceed some threshold. This way the number of spokes or regular features could be readily determined. Counting the times when the data falls below a threshold might also alternatively be used, as well as other methods. The assumption is made that all the wheels of the vehicle have the identical wheel pattern, as is the usual case, particularly in new vehicle advertisement videos. The implementation of the counting of the spokes is very fast as all it requires is scanning a set of digital data points, which can be done very much faster than within one frame of the video. Function 200 could be placed earlier or later within the process flow, and is shown as placed for graphical convenience.

The process then proceeds to 202 to determine the horizontal and vertical dimensions of the designated wheel assembly and wheel. These quantities can be determined by commonly available software that measures the vertical and horizontal extent of the wheel and wheel assembly data relative to the size of a known object. In this case it could be the length of the vehicle, which was estimated previously, or the height of a person or other object in the scene whose size is known or can be approximated. A more accurate determination might be a simple counting of the data elements in an array relative to data from a known object. Alternatively the computer could simply access a lookup table of common wheel assembly sizes for a number of vehicles in various countries, and accept the uncertainty in actual rotation rate that would result from not knowing the exact size, which may be sufficiently accurate for the purposes of this embodiment.

The process then proceeds to 204 which is a computer decision function, similar to that discussed previously in the currently preferred embodiment except for being entirely automated, which compares the dimensions (X) and (Y) of the wheel to determine if they are similar to each other or very different. If the function makes a YES decision it passes the process to computer function 208 for further action, and if NO to computer function 206 to modify the wheel image dimensions so that they become substantially circular.

The criteria used by 204 to make the decision can be fairly loose, because foreshortening is a cosine function of the angle at which a circle is observed, and a few tens of degrees off normal viewing geometry causes little image non-circularity. The opposite is true of course, when viewing a circle close to right angles to the line of sight, in which case the cosine function is rapidly changing. To that end function 206 simply expands the (X) dimension until matches the (Y) dimension, making the dimensions substantially equal. Alternatively the (Y) dimension could be decreased until it matches the (X) dimension, to the same end. Either of these operation would take place in the digital domain and are very fast. The computer could measure the two dimensions and the use the secant of the vertical to expand the horizontal, if desired, to the same end. If the images are small or the video noisy it would be better to invoke the former to avoid noise-induced errors. When complete the process proceeds to 208.

Function 208 determines the actual rotation rate of the wheel assembly over an averaging number of frames, identified as “N” frames, in a similar way as described previously in the currently preferred embodiment. Once the actual rotation rate during the N video frames is known, the process proceeds to function 210 which will select the proper file from a number of pre-calculated and stored files for the apparent rotation rates as a function of the actual rotation rates, choosing the file which was calculated for the now measured number of spokes in the wheel, for the now measured frame rate, and for the now measured wheel assembly diameter. This set of files would be provided by 64 and are thus accessible by or stored in 70. The data on these files would be similar to those of FIGS. 7A and 7B. They could be digitized and stored in the form of look-up tables if desired, or could be calculated as needed by 70 from the basic principles.

Function 210 also determines whether the wheel actual rotation rate as measured by 208 over the chosen “N” frames appears in a “MODIFY” OR “DO NOT MODIFY” region in the chosen chart, corresponding to the example shown in FIG. 7B. The process then proceeds to 220 which is a decision function that passes the process to 232 if the actual rotation rate is such that no image modifications are necessary, and to 222 if modifications will be required. Function 222 chooses an appropriate image substitution wheel modification strategy using similar rationales and choosing from among similar choices that were discussed in previously in the currently preferred embodiment, except that it is automated. The process then passes to 226 which selects a modified wheel image to be used from among the available choices, all in digitized form, in a manner similar to that of discussed previously in the currently preferred embodiment except that it is wholly automated.

The process is then passed to computer decision function 228 to determine whether the wheel images require de-circularization. In order to determine that 228 accesses the decision made by 204 to determine whether the wheel dimensions (X) or (Y) were modified to attain circularity. If the 204 decision was YES then no circularization was done and the process proceeds directly to 232. If it was NO then the dimensions were modified to attain circularity and the process proceeds to 230 to perform the inverse operation and de-circularize the image back to its original shape for insertion of the modified wheel images. To do that it would compress the (X) dimension of the wheel image back to the same dimension that it has before 204 expanded it The process then proceeds to 232 which is a function that holds the data for the wheel assembly and wheel image in the modified frame and measures its dimensions, shape, and size, and adjusts them so that they match the stored dimensions in the original wheel assembly dimensions, should they require any adjustment. In that sense this a “vernier” adjustment to fine-tune the size adjustment and dimensions so that the new images will be seen as seamlessly replacing the original ones when the image replacement is actually performed. The sizing adjustment is a simple scaling in the software, and the criteria for size match are contained in 64.

The process then proceeds to 234 which performs the insertion and replacement of the original wheel assembly images with the modified wheel assembly images into the first attainable targeted frame. The targeted frame was designated by 194 as part of the overall process timing strategy, taking into account all the sources of time delay in the process. The actual modification by 234 of each selected frame of the video with the new images can be accomplished in a number of ways and with criteria similar to those described previously in the currently preferred embodiment except for being wholly automated.

The process then proceeds to 240 which is a decision function with software means to assess whether the timing or quality of the modification, if approved, would meet the criteria laid out in 64. This could consist of simple counting and parity checks to verify that the proper images were sent to the proper target frames. It could also be more complex measurements of the wheel or wheel assembly dimensions as modified in several frames, and comparing them with the original dimensions, and measuring the maximum allowable differences as specified in 64. Function 240 could also be used to start human checks in the setup and troubleshooting phases, and then bypassed for in-line automated operations. Should the automated checks in 240 indicate that the quality is lacking the process can be sent to an optional operator quality check in 236 in which the operator decides whether the modified wheel images prepared by 234 and presented to him were satisfactory or whether rework or new choices are desired, and he proceeds to his decision function 238. If his answer is NO he commands 70 to return the process back to 194 or a later step for a repetition loop. If he decides YES then he commands 70 to pass the process to 242, 244 or 246. This is the same outcome from 240 had the automated quality check resulted in a YES decision.

This operator function is optional. It would be of greatest utility in the set-up, checkout, and initial learning operations of the process, during which operator intervention might be necessary in order to make changes necessary for smooth functioning with minimal delays. It could be expected that once the process is shown to be reliable and with low missed inappropriate rotation recognition and correction rates, that the operator function to start over would be only rarely invoked. Furthermore the functions 238 and 240 have a YES default outcome in which, if the operator has not made a NO decision, the YES outcome is the default decision in order to introduce minimum delay into the process. Furthermore, the computer can be instructed at the outset of installation by the operator, either through 64 or by commands, to deactivate functions 236, 238, and 240 after they are no longer felt to be necessary, in order to reduce system time delays as much as possible.

242 is a sequencing function that coordinates and executes a series of steps and commands 194 in the process to select and designate another wheel in the same vehicle and repeat steps 194-240 as long as the vehicle is visible in the video, skipping steps 198, 200, 210, 222, and 226. This will result in the second wheel's image being inserted in a target frame selected as the first attainable due to reduced number of steps and the now calibrated delays. This process could be continued with 242 choosing alternate wheels in the vehicle so that each gets modified in every other frame. While both wheel images of a vehicle would thus be modified, in time every frame would have one wheel with the original image and the other wheel with the modified image, with less than ideal results. A better alternative would be that the image of the first modified wheel, or the entire vehicle with one modified wheel image, could be put into buffer storage by 232 for one frame and then 234 could insert both the first stored modified wheel image and the second modified wheel image into the second target frame.

That process could be continued alternating which wheel image is stored so that every frame has inserted into it the latest modified image of one wheel and the previous modified image of the other wheel. That way only modified images of wheels are passed to the process output, with every frame having one wheel whose image lags one frame behind the other, the wheels taking turns which one lags the other. Though introducing some extra delay the results would be high quality modified apparent wheel rotation rates as the eye of a viewer cannot detect such rapid changes and construes both wheel images to be continuous.

The continued presence of the vehicle can be assured either by 190, which can continually scan for its presence, or by communications between 242 and 190 which can be coordinated so that the functioning of the process could stop when the vehicle is no longer visible in the original video. However, should another vehicle appear in the video while the first vehicle is still visible this scanning for vehicles can alert 242 which passes the process to and commands 244 to assume essentially the same sequencing function as 242 in parallel so that both vehicles and their wheels are tracked, and the proper wheel images are modified and inserted into the video for the second vehicle as well as for the first by 234.

This implies that there could be up to three modified wheel images in buffer storage at one time, to be introduced along with the most recent modified image for a fourth wheel into the vehicle in the target frame, and that up to every fourth frame, in that case, could contain these modified images. Since the modified images of wheels would only be modified to make them blurred, possibly with one uniform blurriness or perhaps with one that varied somewhat with actual rotation rate, it is doubtful that even having three frames delayed out of four by a maximum of 3/60 of a second would be noticeable in the video by the users. The software implementation for either of these options is similar and can be accomplished readily by any programmer normally skilled in the art.

The process then proceeds to 246 which is an optional function that assembles and accumulates the modified and unmodified frames in the proper sequence of the original video into a temporary video segment, which can optionally be shown to the operator for his approval off-line, during the installation and checkout process, if desired. If so the process proceeds to 248 in which the operator views the temporary video segment prepared by 246 during which he would be able to see the effects of the image modification process. While this may not be very useful′ in actual on-line use because of the very short times available and the very undesirable time delays incurred, it would be extremely useful in the initial familiarization, operational practice, and checkout process, as well as for troubleshooting. If the operator responds in 248 he decides in 250 whether the quality is adequate. If YES, which is the default decision, he does nothing. If NO he commands 70 to send the process to 252.

The output of function 244, and that optionally of 246, represents the new video stream modified as appropriate, and the process proceeds to function 252 which is a fast video switch, implemented either in software or in hardware or both, that has the ability to switch between the original video 62 and the modified video in a time shorter than one video frame. This switching function would be enabled by command from 250 to 70 to switch 252. If the operator decided the quality was not acceptable. If the operator deems the final quality acceptable he does nothing, and the fast video switch 252 will continue to stream the modified video to 254, which is his default outcome in 250. Should 252 be activated it would effectively sidetrack the image modification process until that command is reversed, but without interrupting the video stream which continues to stream the original video to be distributed by 254 to the media and the users/viewers. While the modified video would be sidetracked it could simply be stored in buffer memory and the process kept operating until the decision was made to switch it back to the modified video stream by commanding 70 to switch 252. The inclusion of this operator optional intervention assures that a poor quality product will not make it all the way without a conscious decision by the operator let it do so.

Should the operator determine it advantageous, or there were instructions in 64 regarding trying again after some elapsed time so state, the operator can reverse his decision to stream the original video to 254 by overriding his decision made in 250 and commanding 252 to once again pass the modified video stream to 254. This can be repeated as often as the operator or the instructions of 64 deem it necessary. However, whether so instructed in 64 or by operator decision one or more, or all, of the optional operator assessment and intervention functions can be omitted from the repeat sequences of steps commanded by 242 and 244 and the rest of the process. Thus the process can be limited to a wholly automatic and autonomous process with absolute minimal time delays, that can be “on-line” all the time without operator participation, which is envisioned as the normal mode of this embodiment.

As in the currently preferred embodiment the functioning of the process produces a video of the vehicle or vehicles in which the wheels appear to be rotating at rates appropriate for the vehicle speed, as illustrated in FIGS. 17A-17L for low speeds, or appropriately blurred as illustrated in FIGS. 18A-18F. The output of 252 is a seamless video stream containing either the original or modified video frames in whatever sequence the process attained which is distributed to the media and users in the usual manner of conventional streaming video, and will assure that the viewers will be unaware of the modifications, whether inserted or not.

While the steps of the process in this embodiment will accomplish its objectives, some steps may not be necessary and might be omitted, while some may be combined into fewer steps. An example is the omitting of steps 204, 206, 228, and 230 which made the wheel image circular for easier measurements and operations, used a circular modified image, and then made it non-circular again to match the original wheel shape. An option exists to perform all operations on the non-circular wheel images, which will be elliptical, since the are software programs that can perform the counting of the spokes and wheel sizing operations on elliptical images rather than on circular ones. While this would save some complexity and time delay there are limits to the ellipticity that can be accommodated in such operations due to the inherent noise and inaccuracy involved in measurements on small quantities, which could decrease the accuracy of the image substitution process, and thus such techniques could be used providing the angle of viewing of the wheel with respect to its axis is not close to zero or 180 degrees.

Description of the Second Alternate Preferred Embodiment

This embodiment is a method and apparatus of a human-controlled and machine implemented process which is designed to operate on recorded video media off-line and not in real time, and the resultant new recorded video to be distributed to media which would stream it to viewers. This embodiment corrects rotation rate of the wheels of vehicles in a video which are perceived in user displays as being inappropriate for the actual vehicle speed, replacing them with images whose rotation either cannot be perceived by eye while leaving unchanged the images of wheels whose rotation rate is perceived by eye to be appropriate for the actual vehicle speed. This embodiment is similar to that of the currently preferred embodiment except that it employs vehicle speed estimation by the operator rather than its measurement, which allows fewer steps and less complexity at the cost of reduced accuracy. This embodiment addresses the wheels of a vehicle and multiple vehicles in a video serially.

Figures in this embodiment: FIGS. 5A through 21 (previous embodiments) and 22-24.

-   256 Identification of a vehicle and its wheels, and their     designation -   258 Supplying video frames, video, and auto tracking -   260 Identifying the time boundaries between proper and improper     wheel rotation -   262 Estimation of the vehicle speed at the boundaries of blurred     image perception -   264 Calculation of the actual rotation rate at the boundaries, and     storage of data -   266 Designation of time ranges to be modified, and choosing one -   268 Selection of the best image modification type -   270 Selection of the frames, and their adjustment and storage -   272 Formatting and adjustment of the selected modification frames to     match their originals -   274 Checking for size and shape fit of the selected wheel images     into their originals -   276 Decision as the quality -   278 Formatting all frames in the selected time region -   280 Insertion of modified video images into the video time range,     frame by frame -   282 Preparation of a temporary video of the modified selected time     range, and its storage -   284 Operator observation of the temporary video -   286 decision of attained quality -   288 Repetition of some process steps is required -   290 Returning to a previous step and choosing a different time range     to modify -   292 Repetition of previous process steps for all frames in the time     range, and storage -   294 Returning to previous process step and choosing a different     wheel in the same vehicle -   296 Repetition of previous process steps for the different wheel -   298 Returning to a previous process step and choosing a different     vehicle should it appear -   300 Repetition of previous process steps for the different vehicle -   302 Accumulation and assembly of all modified and unmodified frames     into a new video -   304 Operator observation of the new video -   306 Final quality decision -   308 Recording the new video and its distribution to media

The process for this embodiment begins in FIG. 22 with functions 62-68, as does the currently preferred embodiment. Once they are completed the operator, in function 256, views the video or some portion thereof, either directly from 62 or from 70, identifying the first vehicle appearance to be addressed and its wheels, and manually designating them to 70. This appearance of the vehicle in the video is illustrated in FIG. 10, which shows an example of a video, its start as time A, the appearance of a vehicle noted as at time B and its disappearance noted as at time C and identified d as 96, a second vehicle appearance at time D and its disappearance as time E and identified as 98, and the ending of the video in time F. This description continues referring to the vehicle as though there were only one vehicle making an appearance in the video, but it will be understood that the process applies to all appearances by one vehicle and also to all appearances of any other vehicles, should there be more than one. Thus 96 could be the first appearance of vehicle #1 and 98 its second appearance, as an example.

Although the operator can choose any point in the video to start the process proper he could advantageously choose a point in time in which a vehicle makes its first appearance in the video, such as point A in FIG. 9. Alternatively he could choose a point in time at which he first perceives the wheels of the vehicle to have an inappropriate rotation rate for the vehicle apparent speed. The operator designates not just the presence of a vehicle but actually designates the vehicle itself, and one or both of its visible wheels via their coordinates, distance from the edges, of the video frames, or specific characteristics of their image that would allow the computer to track them. Function 258 can also supply images, portions of images, frozen video frames, or other information as requested by 256 for display.

The process then proceeds to 260 in which the computer, on command from the operator, plays the video again on 68, and the operator identifies the beginning and ending times in the video during which the apparent wheel rotation rate appears inappropriate for the apparent vehicle speed and must be modified, and designates them to 70. If he sees any wheel rotation rates that appear to be above the level L1 or L3, referring to the actual/apparent wheel graphs or data stored in 64 as in the previous embodiments, and are somewhat, mostly, or totally blurred, he will designate them to 70 as “not needing modification”. However if he sees wheels rotating or stopped and the spokes are clearly visible he will need additional information in order to unambiguously determine whether they require modification or not because they could be caused by ambiguities due to actual rotation rates near the sub synchronous points or higher. He therefore must then estimate the vehicle speed, even if crudely, to resolve these ambiguities.

This additional information he will obtain from observing the moving vehicle against the background of the scene in the video, and making an approximate estimate of the vehicle's speed over the ground. If he judges the vehicle's speed to be below, for example, about 38 kph or 24 mph, which corresponds to an L1 or L3 being below about 5 revolutions per second from TRR=V/πD where V is the vehicle velocity in kilometers per hour, and D is the wheel assembly diameter in meters and assumed at 0.67 meters, and he perceives the wheel to be rotating forward and with the spoke pattern visible, he will deduce that the wheel actual rotation rate is on line 34 in FIG. 7A, and thus in the TRR-0 to TRR-1 range in FIG. 7B, and therefore in the “do not modify” range. If, however, he judges the vehicle speed to be substantially above 38 kph or 24 mph in this example then the distinct rotating spoke pattern he observes means that the actual rotation rate must be higher than TRR-1 and therefore near one of the sub synchronous points, the synchronous point, or even a super synchronous point, and he will therefore designate the rotation rate at that time as inappropriate and one that is in the “modify” range. In this way the operator designates to 70 which time ranges, and therefore implicitly which frames, need to be modified and selects the first such range for action.

In this embodiment the operator has insufficient information to take any action other than that shown in FIG. 8A as line 34 for actual rotation rates below about 10 revolutions per second in a first category, and line 76 for all higher actual rotation rates in a second category, the approximate level of the actual rotation rate being estimated by the vehicle speed divided by the product of pi and the wheel assembly diameter. The operator command to 70 is simply that if the frame is in the first category the wheel is not to be modified, but if it is in the second category the wheel image is to be modified by a blurred image, and that the blurred image is to be the same for all such frames. The operator next commands 70 to proceed to 264 to calculate the wheel actual rotation rate at each of the boundaries he has identified using his estimated vehicle speed and the wheel assembly diameter, and store them.

The computer can calculate all these off line and store them while the process proceeds to 266, in which the operator views the video and designates the time ranges to be modified and selects the first one to address in commands to 70, and the process proceeds to 268 in FIG. 23 which the operator selects the most appropriate wheel modification image that would be inserted into the original wheel in the ‘modify” frames of the selected time range, from the stored image data from 64. He then proceeds to 270 in which he commands 70 to step through the selected time range, at an accelerated pace if he wishes, so that he can identify and designate a number of frames that he selects to represent a sampling of times when the foreshortening of the wheels appears to be changing. As an example he might select a frame every 10 seconds if the foreshortening of the wheels appears fairly constant, and a frame every second in times when the foreshortening appears to be rapidly changing.

The operator then commands 70 to proceed to 272 which takes the selected modified wheel image type from 268 and adjusts it to match the size and shape of the wheel images in the identified frames in the original video, which were all within the selected time range for modification, and stores the resulting modified wheel images. The process then proceeds to 274 in which the operator steps through some or all of the selected frames by command to 70, and checks whether the replacement size and shape match well enough the size and shape of the original wheel images, within the guidelines of 64 or his own judgment, and makes a decision in 276 as to their quality. If not satisfactory he commands 70 to send the process back to repeat 272, and if still not satisfactory to 270. If he judges the quality of most of the selected frames adequate he commands 70 to send the process to 278.

Function 278 formats all the modified wheel images in the designated time range frame by frame, adjusts each one to match the size and shape of the wheel images in the original video, and stores them. The process then proceeds to 280 which performs the actual insertion of the now modified and formatted wheel images into the original video frames, one by one, in proper sequence, and stores them. The process then proceeds to 282 which assembles the modified frames into a temporary video, stores it, and proceeds to 284 in which the temporary video is shown to the operator. If the operator deems the quality adequate in his decision function 286 he commands 70 to proceed to 290. If the quality is lacking he commands 70 to send the process to 288 to repeat steps 262 to 286 one time, and if still lacking to revert to 256 or abort to start anew. If the quality is satisfactory the operator commands 70 to proceed to 290 in FIG. 24 and return the process to 268 to choose another time range to modify and then to 292 to repeat steps 268-290 for all the frames in the new time range and store them. This is repeated until all the time ranges to be modified in the vehicle appearance are modified.

The process then proceeds to 294 in which the operator commands 70 to return to 266 and choose a different wheel in the same vehicle and then to 296 to repeat steps 266-294 for all frames in the new time range, and store them. The process then proceeds to 298 in which the operator commands 70 to return to 266 to choose a different vehicle appearance, if any, and then to 300 to repeat steps 266-298 for another vehicle appearance. The operator then commands 70 to proceed to 302 which assembles all the accumulated and stored modified frames and the unmodified frames from the previous functions and assembles them in proper order into a temporary new video. This video is shown to the operator in 304 and he observes the quality of the new video in its totality. If he deems the quality acceptable, using his criteria 64 or his own judgment, he decides YES in his decision function 306, which commands 70 to pass to 308 to make one or more recordings of the modified video stream in one or more media types and formats, which are then ready to be made available for distribution to viewers instead of the original video 62.

The recording(s) can be made as software, if such is the medium to be distributed, or if hard media are desired by any machine capable of that task, whether it includes 70 or is simply turned on by 70. If the operator deems the final quality unacceptable he can command 70 to return the process back to 256 or another step to repeat the process, or simply abort. Whereas the default decisions of 276 and 286 are YES so that the process can be given more chances to perform as intended, the default decision in 306 should be NO so that a poor quality product will not make it all the way without a conscious decision by the operator to do so.

The results of the operation of this embodiment will be wheels perceived to be correctly rotating for every time range when the vehicle speed is low, as shown in FIGS. 17A-17L, in which a wheel is seen as rotating appropriately for its actual rotation rate. When the actual rotation rate is not low, that is when it is above TRR-1 in FIGS. 7A-7B, the wheel images are always deliberately blurred so as to prevent the spokes or other patterns to be perceived as rotating at all, which is appropriate at speeds above the blur limits of L1 or L3 in FIGS. 7A and 7B.

Thus the inappropriate apparent rotation of the wheels will have been remedied by image substitution where appropriate for all wheel actual rotation rates and therefore for all vehicle speeds, and the undesirable effect of the stroboscopic effect will have been completely eliminated; the image modifications and substitutions are undetectable by the viewer; and the process has produced an entirely new and unexpected result: the absence of apparent inappropriate rotation rate of wheels that do not correspond to actual vehicle speed as seen in video displays of moving vehicles.

Detailed Description of the Third Alternate Embodiment

This embodiment is a method and apparatus of a fully automated process implemented on a computer, with optional and limited human control of the computer, which is designed to operate in near real time and in-line with streaming video services. This embodiment is similar to the first alternate embodiment but adds parallel channels of processing and appropriate sequencing so that both visible wheels of a vehicle and two or more different vehicles can be processed simultaneously in order to reduce time delays to an absolute minimum.

Figures in this embodiment: FIGS. 5A through 24 (previous embodiments) and 25-26

-   310 Identification, designation, and auto tracking of vehicles and     their wheels -   312 Visual identification of vehicles and manual designation, or     process start -   314 Select timing strategy and set the M frames and target frame -   316 provision of image rotation and measurement resources -   318 Determination of vehicle speed, direction, and angles -   320 Determination of frame rate and counting number of spokes in     wheel -   322 Determination of the vertical and horizontal dimensions of a     wheel -   324 Deciding as to whether the two dimensions are about equal -   326 Expansion of the horizontal dimension to equal the vertical     dimension -   328 Determining the wheel actual rotation rate over a set of N     frames -   330 Determining if frames are in a “modify” region of apparent/true     rotation rates graph -   332 Deciding whether the frames require modification -   334 Choosing an appropriate image substitution strategy -   336 selection of a modified wheel image from the available choices -   338 Optional human review of the choice made -   339 Deciding whether a previous decision was YES -   340 Compression of the horizontal dimension back to its original     dimensions -   342 Adjusting the size and fit of the modified image -   344 Sequencing and buffering for the modified wheel images -   346 Insertion of the modified wheel images into the new target frame     and store -   348 Deciding on the quality and timing of the substitutions -   350 Optional visual quality check of wheel images -   352 Decision as to the quality of the images -   354 Optional assembling the frames into a new video -   356 Optional final operator quality check -   358 Deciding whether to pass the modified video stream or not -   360 Providing fast video switch -   362 Distribution of either the original or the modified video stream     to the users

The process begins with computer function 310 in FIG. 25 with the computer software accessing the start of the video 62, which is illustrated in FIG. 10 as point A. 310 then autonomously identifies, designates, and begins tracking the first vehicle's first appearance in the video, such as shown in the illustration of FIG. 10 as the point B of the vehicle's appearance, 96. The software definitions and use provisions and discussions in the currently preferred embodiment are fully applicable to this embodiment.

An option is provided in function 312 for the operator to identify and designate the vehicle to 70 while viewing the video 62 on 68 as the first step in the process, and is provided as insurance against the process having unexpected difficulties at its outset. Alternatively the operator can simply initiate the machine process by a command to 70 to “start” 312. The operator could also initiate the process by an even simpler switch actuation which would power or alert 70 to the fact that its functions are now desired, so as to prevent it from operating all the time on a video feed and rather only when and if so desired. Nonetheless, as the process is to be automated so as to minimize processing time delays, it is anticipated that the process starting with 310 be active all the time when functioning. This could be implemented by 312 once the automated vehicle recognition function works reliably enough and has a low inappropriate start rate, as would be determined by analysis, experience, or both.

Once the process is started 310 begins observing the video image stream to detect and identify the first appearance by a vehicle in the video stream, following the instructions, information, and criteria of 64, and designates the vehicle by a unique identifier code. The function then similarly detects, identifies, and designates its wheel, or both wheels should they both be visible, and begins tracking the vehicle and each wheel separately and assigns each a unique code. Function 310 also continually observes the video stream continuously to detect when the vehicle disappears, or when a different vehicle makes its appearance, shown as point D and curve 98 in FIG. 10, and if so it designates the second vehicle by a unique identifier code, detects and designates its wheels, and autotracks them, and repeats the process for any additional vehicles that make their appearance in the video. 310 could thus track several vehicles and their wheels simultaneously, an important factor since the two visible wheels of any vehicle may have a different shape when it is turning, and since the other vehicle or vehicles may make appearances simultaneously with the first vehicle.

310 also acts as a master assignment function that selects and sets the timing and channel strategy for the embodiment. It assigns the first vehicle and its wheels to one of two or more parallel processing channels, as indicated in the flow diagram, beginning with function 314 for vehicle #1. Though all vehicle channels will have the same functions and functional step numbers and will operate in parallel, the flow diagrams of FIGS. 25 and 26 indicate the presence of the additional and separate channels but do not show their detailed functions or flow. The description and operation of the parallel channels will not be further addressed until function 344 is described, as they simply mirror all descriptions of the described channel but applied to other vehicles and their wheels.

Function 314 selects the timing and strategy for setting the M frames and the target frame, similarly to the operations described previously in of the first alternate embodiment. The designation of the wheels and the measurement of the vehicle speed could start in parallel as they are not dependent on each other, and the measurement of the number of spokes could start independently also. The counting of the number of spokes can also occur in parallel with the vehicle speed measurement, but the wheel dimension determination must follow the vehicle speed determination, and the system total processing time must necessarily be additive to all the other functional times, so that the total time could be reduced to T₂ in FIG. 20C by paralleling some functional steps. In addition, once the process has stepped though and modified the first frame image, the process for doing so the following frame images can be considerably shorter as long as it is within the same vehicle appearance. This is because only the vehicle speed and the wheel dimensions must be measured, and with the unavoidable system time delay the total delay could be shortened to T₃ in FIG. 20D. The functioning of 314 requires that the total system time delay be measured and stored, which would be done by the equipment manufacturer, and calibrated in the installation by the operator during the set-up process, and would be similar though not necessarily identical for each parallel channel processing the two vehicles.

The process then proceeds to 318 to determine the vehicle speed, which will be needed in order to determine the wheel actual rotation rate, which in turn is necessary for the image substitution process. There are two separate process flows for the different visible wheels on a vehicle, which begins with 318 and ends in 342, the assignment of which channel handles which wheel being handled by 310. The description and operation of each of the parallel channels in this embodiment, beginning with function 320 through 342, is similar to that of one channel in the first alternate embodiment, functions 200-230 described previously, and thus that text will not be repeated here, except that 318 has a second important function which is to assign each wheel being tracked to one of two parallel channels. These are indicated in FIG. 25 and will operate in parallel, one for each of the two visible wheels of the vehicle.

The process description resumes with 344 which is a sequencing and buffering function that collects the modified wheel images from up to two wheels in each of two or more different vehicles from the separate processing flows that originated in 314 and 320. Since these images may each have a different total delay they may arrive at 344 at slightly different times, and thus they are each placed into buffer storage and collected. When all have arrived they are sent to 346 which performs the simultaneous insertion and replacement of all the original wheel assembly images in all visible vehicles with the individual appropriately modified wheel assembly images into the next possible frame, whether the original targeted frame or a slightly delayed frame should the parallel processing requirements result in slightly mire time delay.

The rest of the process beginning with the actual modification by 346 of each selected frame of the video with the new images and ending with the distribution of the video stream to the users is similar to that described for the first alternate embodiment, with the principal difference that the sequencing and looping functions that were required in the first alternate embodiment, because all wheels and vehicles were processes serially, are not required in this embodiment due to the implementation of parallel processing channels. Thus the text of the noted paragraphs in the first alternate embodiment will not be repeated here, though the process flow diagrams of FIGS. 25 and 26 are shown in their entirety.

Summary of Advantages

The embodiments described provide apparatus and method which will result in the apparent rotation rates of the images of wheels in moving vehicles as presented in television, video, movie, computer, or other media displays to be perceived as appropriate for the actual vehicle speeds in the displays; provide similar apparatus and method which will result in the apparent translation rates of linear objects containing regular features as presented in television, video, movie, computer, or other media displays to be perceived as appropriate for the actual linear motion of those objects in the displays; generally make much more satisfying the television, video, movie, computer, or other media viewing experience when containing moving objects with regularly placed areas or features; generally make more pleasing the experience and much more attractive the vehicles shown in these media for advertising or other purposes and resulting in increased sales, greater viewer approval, or both; remove any limits in current vehicle imaging such as restrictions in showing only some viewing angles, situations, etc. which may be imposed in order to minimize the stroboscopic effect, which limits or complicates or makes more expensive the production of videos in any media for advertising and other purposes; have low cost and fairly rapid implementation, since all embodiments employ mostly existing software and hardware; ensure that operation of the implementation will also be low cost because most of the functions of the method can be performed by commonly available but specific software running on a widely available general purpose or special purpose computer; or not available is simple enough that such can be readily programmed by those with ordinary skill in the art. The embodiments differ principally in the degree and kind of human operator interaction with, and oversight of, the machine computer carrying out the process, and oversight of the final product; and whether they operate off-line on a video and produce another recorded video, or in-line with the video while it is being streamed to viewers.

CONCLUSIONS, RAMIFICATIONS, AND SCOPE

Each of the embodiments of the invention presented can either completely eliminate or greatly reduce the perception of inappropriately rotating wheels on moving vehicles which are commonly observed in video, television, computer, movies, video discs or tapes, DVDs, video CDs and other displays. All embodiments are also applicable to all imaging media which show images in a set of sequences, including movie film when converted to electronic presentation or display, and for viewing on analog as well as digital displays or monitors, as well as cellphone displays, tablet computer displays, and the like.

Two of the embodiments are designed to function in a human-controlled and computer-executed mode, applicable to off-line operation in which the operator and process can take all the time necessary to achieve a satisfactory product; and in which the product is another video recording identical to the original source but with the wheel images that appear to be inappropriately rotating for the vehicle speeds when viewed on displays replaced by wheel images that now appear to be appropriately rotating for the vehicle speeds. The other two embodiments are designed to function in a wholly automated computer-executed mode, with optional and limited human participation, and are applicable to streaming video data which are modified with the same result as in the first mode but with minimal time delays and while being streamed to the user displays in real time or near-real time, and are intended to be on-line with distribution channels to viewers and others all the time and to function in automatic mode; but their operator controlled mode allows for a period of manual installation and checkout, and for manual troubleshooting.

The appearance of seeing the wheels of moving vehicles in videos without the inappropriate rotation effects caused by the stroboscopic effect would have the double advantage of producing a more pleasing and less disconcerting experience for the viewing public, and being a boon to vehicle manufacturers and advertisers, in the latter case saving money and selling more vehicles.

All embodiments are applicable regardless of the frame rate, the interlace rate, the scan rate, the progressive scan rate, film projection rate, or other rate at which the images are presented in rapid sequence to the display units and thence to the viewers. All the embodiments can function well whether the vehicles and their wheels are viewed normal to the axis of rotation of the wheels or at an oblique angle, as the desired effectiveness of the embodiments can be obtained mostly regardless of such angle. The rotational rates, frame rates, apparent rotational rates, frame rate variations, and other aspects of a moving vehicle presented in television or other video media are illustrated and discussed for example only, and the embodiments would function as well for other values. The embodiments are also wholly applicable whether the vehicles shown in the media are cars, bicycles, motorcycles, trucks, trains, wagons, horse-drawn carts or carriages, aircraft taxiing, or indeed any mobile conveyance with rotating visible parts whose design or construction is such that a series of regular features or marks appear thereon, as opposed to a homogeneous surface or appearance. The embodiments may have applications in industry, government, or entertainment regardless of the parameter values of examples discussed.

It will be also appreciated that it may not be necessary to make the final wheel image apparent rotation rates exactly the proper ones, but that due to the limitations of the human eye and thresholds below which the inappropriate motion may not be observable or perceived as objectionable some latitude may exist in presenting images whose apparent rotation rates, while not perfect, still appear much better if not totally appropriate than the backwardly rotation, stopped rotation, and other apparent rates grossly differing from the proper ones commonly seen in videos, movies, and television.

It must also be recognized that the stroboscopic effect as treated for rotating objects is not limited to objects that rotate, but in general also applies to objects that have regular features and rectilinear or other motion at an angle that is not parallel to the main direction of some or all of those features. This is because if an extended object with regular features, such as a long ladder with a number of steps or rungs for example, is moving in a direction such as being carried by two workmen across the field of view of the video camera, the stroboscopic effect interacting with the frame rate of the viewers' display can cause the horizontal position of the steps of the ladder to be perceived improperly as moving forward, backward, or standing still. This is completely analogous to the improperly perceived rotating wheel assemblies. Therefore the features, characteristics, and methods of all embodiments of this invention must be recognized to apply fully for use with objects of any shape and having rectilinear as well as rotational motion, or combinations of both, so long as they contain elements of their construction or design that have regular features, and whose placement in the video images is varying with time.

The currently preferred embodiment and each of the alternate embodiments are equally capable and worthy, and their presentation and order should not be construed as implying that any has more advantages than the others.

While the embodiment descriptions contain many details and specifics, these should not be construed as limiting the scope of the embodiments, but as merely providing illustrations of several possible embodiments. Thus the scope of the embodiments should be determined by the appended claims and their legal equivalents, rather than by the examples given. 

I claim:
 1. A method of modifying the apparent rotation rate of at least one vehicle wheel containing distinct regularly placed areas, in video media, comprising: accessing a sequence of source video images which include the at least one vehicle wheel, wherein the sequence has a frame rate in frames per second, and wherein the at least one vehicle wheel contains a number of distinct regularly placed areas, is coupled to a vehicle moving at a vehicle velocity, has an overall diameter including tires, and has a rotation rate; providing replacement means to replace only wheel areas of each image in the sequence with modified wheel areas, leaving all non-wheel areas of the images unmodified, wherein a wheel area is an area of an image in which a wheel appears, and a modified wheel area is an area of an image in which a modified wheel appears; setting a first value equal to the frame rate divided by the number of distinct regularly placed areas in a single wheel; setting a second value equal to two times a third value representing a human upper rotational motion perception limit; setting a fourth value equal to the vehicle velocity divided by the product of pi and the overall wheel diameter; wherein when the first value is greater than the second value, operating the replacement means whenever the fourth value is greater than the third value in order to replace the wheel areas of each image with modified wheel areas that are rotationally uniform and indistinct so as to appear blurred or unresolved regardless of their rotation rate when viewed in a video display, and wherein when the first value is less than the second value, operating the replacement means to replace the wheel areas of each image with modified wheel areas having at least one of a first and a second property, the first property comprising having a smaller number of distinct regularly placed areas than the number of distinct regularly placed areas in the source video images, and the second property comprising rotating at a rotation rate of the fourth value when viewed on one or more video displays whenever the rotation rate is less than the third value; and producing a new sequence of video images comprising the source video images and the modified wheel images.
 2. The method of claim 1, further comprising remove any foreshortening for ease in image manipulation, or applying foreshortening after image manipulation for image insertion into the video stream, based on a horizontal and vertical dimension of a vehicle wheel.
 3. The method of claim 1, further comprising producing and storing the new sequence of video images in a time sequence that matches the source sequence.
 4. The method of claim 1, further comprising streaming the new sequence of video images to media in real time for display on at least one display device.
 5. The method of claim 4, further comprising accessing and modifying in parallel the wheel images of a plurality of vehicles in one or more scenes of the source video simultaneously.
 6. The method of claim 5, further comprising buffering and sequencing a plurality of modified wheel areas from the plurality of vehicles for simultaneous and seamless replacement into the new video sequence. 